German Traffic Sign Classifier using a LeNet based Convolutional Network

Pipeline Approach

In this project, I'm building pipelines to classify the traffic signs from the German Traffic Sign Benchmarks.

I'm using the scikit-learn's pipeline framework in order to train the model with various combinations of transformations and estimators.

The starting model is a convolutional network based on the LeNet architecture by Yann LeCun. LeNet was originally designed for handwritten and machine-printed character recognition.

The project is explained in the following sections.

  • Data Exploratory Analysis
  • Model Implementation
  • Pipeline Implementation
  • Preprocessing
  • Model Improvement
  • Testing and Results

Data Exploratory Analysis

The traffic sign images were taken from the German Traffic Sign Benchmarks.

> mkdir data

> # use wget or curl 
> wget http://benchmark.ini.rub.de/Dataset/GTSRB_Final_Training_Images.zip
> unzip GTSRB_Final_Training_Images.zip
> mv GTSRB/Final_Training data/

> wget http://benchmark.ini.rub.de/Dataset/GTSRB_Final_Test_Images.zip
> unzip GTSRB_Final_Test_Images.zip
> mv GTSRB/Final_Test/ data/

> wget http://benchmark.ini.rub.de/Dataset/GTSRB_Final_Test_GT.zip
> unzip GTSRB_Final_Test_GT.zip
> mv GT-final_test.csv data/Final_Test/Images/

Training Data Folder Structure

The training images are organized in folders by category. Each folder is meant for one category (i.e. stop sign) and has a label file (.csv) which is actually semi-colon delimited (not comma delimited).

data
 + Final_Training
    + Images
        + 00000
            + 00000_00000.ppm
            + 00000_00001.ppm
            ...    
            + GT-00000.csv
        + 00001
            + 00000_00000.ppm
            + 00000_00001.ppm
            ...    
            + GT-00001.csv
        ...

All images are stored in the PPM format (Portable Pixmap, P6). You'll need to install both matplotlib and pillow to handle such image format. If you use one of the evironments yml files in this repository, this will be taken care of.

Import Required Libraries

In [1]:
import glob
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import cv2
import os
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import tensorflow as tf
import tensorflow as tf
from tensorflow.contrib.layers import flatten
from pipeline import NeuralNetwork, make_adam, Session, build_pipeline

matplotlib.style.use('ggplot')
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

Loading Training Data

All train image paths are combined into one dataframe for convenience.

In [2]:
TRAIN_IMAGE_DIR = 'data/Final_Training/Images'

dfs = []
for train_file in glob.glob(os.path.join(TRAIN_IMAGE_DIR, '*/GT-*.csv')):
    folder = train_file.split('/')[3]
    df = pd.read_csv(train_file, sep=';')
    df['Filename'] = df['Filename'].apply(lambda x: os.path.join(TRAIN_IMAGE_DIR, folder, x))
    dfs.append(df)
In [3]:
train_df = pd.concat(dfs, ignore_index=True)
train_df.head()
Out[3]:
Filename Width Height Roi.X1 Roi.Y1 Roi.X2 Roi.Y2 ClassId
0 data/Final_Training/Images/00000/00000_00000.ppm 29 30 5 6 24 25 0
1 data/Final_Training/Images/00000/00000_00001.ppm 30 30 5 5 25 25 0
2 data/Final_Training/Images/00000/00000_00002.ppm 30 30 5 5 25 25 0
3 data/Final_Training/Images/00000/00000_00003.ppm 31 31 5 5 26 26 0
4 data/Final_Training/Images/00000/00000_00004.ppm 30 32 5 6 25 26 0

Annotation format

  • Filename: Filename of corresponding image
  • Width: Width of the image
  • Height: Height of the image
  • ROI.x1: X-coordinate of top-left corner of traffic sign bounding box
  • ROI.y1: Y-coordinate of top-left corner of traffic sign bounding box
  • ROI.x2: X-coordinate of bottom-right corner of traffic sign bounding box
  • ROI.y2: Y-coordinate of bottom-right corner of traffic sign bounding box
  • ClassId: Assigned class label

The following points are worth mentioning:

  • Image sizes are not fixed
  • Sign sizes are not fixed
  • Sign's center location is not fixed

Later on, I'll examine sample images to further clarify those points.

Class Distribution

There are 43 traffic sign classes in 39,209 training images.

In [4]:
N_CLASSES = np.unique(train_df['ClassId']).size  # keep this for later

print("Number of training images : {:>5}".format(train_df.shape[0]))
print("Number of classes         : {:>5}".format(N_CLASSES))
Number of training images : 39209
Number of classes         :    43

The distribution of classes are very skewed.

In [5]:
def show_class_distribution(classIDs, title):
    """
    Plot the traffic sign class distribution
    """
    plt.figure(figsize=(15, 5))
    plt.title('Class ID distribution for {}'.format(title))
    plt.hist(classIDs, bins=N_CLASSES)
    plt.show()
In [6]:
show_class_distribution(train_df['ClassId'], 'Train Data')

The name of each sign are stored in sign_names.csv file. We can use it see the distribution per sign names.

In [7]:
sign_name_df = pd.read_csv('sign_names.csv', index_col='ClassId')
sign_name_df.head()
Out[7]:
SignName
ClassId
0 Speed limit (20km/h)
1 Speed limit (30km/h)
2 Speed limit (50km/h)
3 Speed limit (60km/h)
4 Speed limit (70km/h)
In [8]:
sign_name_df['Occurence'] = [sum(train_df['ClassId']==c) for c in range(N_CLASSES)]
sign_name_df.sort_values('Occurence', ascending=False)
Out[8]:
SignName Occurence
ClassId
2 Speed limit (50km/h) 2250
1 Speed limit (30km/h) 2220
13 Yield 2160
12 Priority road 2100
38 Keep right 2070
10 No passing for vehicles over 3.5 metric tons 2010
4 Speed limit (70km/h) 1980
5 Speed limit (80km/h) 1860
25 Road work 1500
9 No passing 1470
7 Speed limit (100km/h) 1440
3 Speed limit (60km/h) 1410
8 Speed limit (120km/h) 1410
11 Right-of-way at the next intersection 1320
35 Ahead only 1200
18 General caution 1200
17 No entry 1110
31 Wild animals crossing 780
14 Stop 780
33 Turn right ahead 689
15 No vehicles 630
26 Traffic signals 600
28 Children crossing 540
23 Slippery road 510
30 Beware of ice/snow 450
16 Vehicles over 3.5 metric tons prohibited 420
34 Turn left ahead 420
6 End of speed limit (80km/h) 420
36 Go straight or right 390
22 Bumpy road 390
40 Roundabout mandatory 360
20 Dangerous curve to the right 360
21 Double curve 330
39 Keep left 300
29 Bicycles crossing 270
24 Road narrows on the right 270
41 End of no passing 240
42 End of no passing by vehicles over 3.5 metric ... 240
32 End of all speed and passing limits 240
27 Pedestrians 240
37 Go straight or left 210
19 Dangerous curve to the left 210
0 Speed limit (20km/h) 210

The following constant is defined for later use.

In [9]:
SIGN_NAMES = sign_name_df.SignName.values
SIGN_NAMES[2]
Out[9]:
'Speed limit (50km/h)'

Sample Images

Let's examine some random images:

In [10]:
def load_image(image_file):
    """
    Read image file into numpy array (RGB)
    """
    return plt.imread(image_file)
In [11]:
def get_samples(image_data, num_samples, class_id=None):
    """
    Randomly select image filenames and their class IDs
    """
    if class_id is not None:
        image_data = image_data[image_data['ClassId']==class_id]
    indices = np.random.choice(image_data.shape[0], size=num_samples, replace=False)
    return image_data.iloc[indices][['Filename', 'ClassId']].values
In [12]:
def show_images(image_data, cols=5, sign_names=None, show_shape=False, func=None):
    """
    Given a list of image file paths, load images and show them.
    """
    num_images = len(image_data)
    rows = num_images//cols
    plt.figure(figsize=(cols*3,rows*2.5))
    for i, (image_file, label) in enumerate(image_data):
        image = load_image(image_file)
        if func is not None:
            image = func(image)
        plt.subplot(rows, cols, i+1)
        plt.imshow(image)
        if sign_names is not None:
            plt.text(0, 0, '{}: {}'.format(label, sign_names[label]), color='k',backgroundcolor='c', fontsize=8)        
        if show_shape:
            plt.text(0, image.shape[0], '{}'.format(image.shape), color='k',backgroundcolor='y', fontsize=8)        
        plt.xticks([])
        plt.yticks([])
    plt.show()

The below are 20 random sample images from the train set.

In [13]:
sample_data = get_samples(train_df, 20)
show_images(sample_data, sign_names=SIGN_NAMES, show_shape=True)
In [14]:
print(SIGN_NAMES[2])
show_images(get_samples(train_df, 100, class_id=2), cols=20, show_shape=True)
Speed limit (50km/h)

Looking at the sample images, the following image characteristics are confirmed:

  • The images comes in different sizes
  • The darkness / brightness of those images are fairly random
  • The images may be slightly rotated
  • The images may not be facing straight
  • The images may not be exactly centered

The first point will be handled in the image pre-processing, and the remaining points will be handled in the image augmentation.

Train and Validation Data set

Train and validation data set are created from the training data.

In [15]:
X = train_df['Filename'].values
y = train_df['ClassId'].values

print('X data', len(X))
X data 39209
In [16]:
X_train, X_valid, y_train, y_valid = train_test_split(X, y, stratify=y, test_size=8000, random_state=0)

print('X_train:', len(X_train))
print('X_valid:', len(X_valid))
X_train: 31209
X_valid: 8000

Model Implementation

Model Architecture

The model is based on LeNet by Yann LeCun. It is a convolutional neural network designed to recognize visual patterns directly from pixel images with minimal preprocessing. It can handle hand-written characters very well.

LeNet

Source: http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

  • The inputs are 32x32 (Black-White 1 channel) images
  • LeNet uses 2x2 sub-sampling (valid padding, max pooling) in convolutional layers (no dropout)
  • LeNet uses a sigmoid squashing function - a scaled hyperbolic tangent: $Atanh(Sa)$ where A=1.7159 and S=2/3
  • LeNet has one 7x12 bitmap for each class, which is a label. There are 10 classes (the digits '0' - '9')
  • 7x12 = 84 which is why the output is 84
  • The output is compared with all the labels (bitmaps) to calculate the error
  • The class with the smallest error is an estimated digit value

Our model is adapted from the LeNet as follows.

  • The inputs are 32x32 (RGB - 3 channels) images
  • The activation function is ReLU except for the output layer which uses Softmax
  • The output has 43 classes
Layer Shape
Input 32x32x3
Convolution (valid, 5x5x6) 28x28x6
Max Pooling (valid, 2x2) 14x14x6
Activation (ReLU) 14x14x6
Convolution (valid, 5x5x16) 10x10x16
Max Pooling (valid, 2x2) 5x5x16
Activation (ReLU) 5x5x16
Flatten 400
Dense 120
Activation (ReLU) 120
Dense 43
Activation (Softmax) 43

Model Implementation

The NeuralNetwork class is defined to provide common operations in neural network using Tensorflow. See the network.py for details.

The first network (based on LeNet) is defined as follows:

In [17]:
INPUT_SHAPE = (32, 32, 3)

def make_network1(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 6])
            .max_pool()
            .relu()
            .conv([5, 5, 16])
            .max_pool()
            .relu()
            .flatten()
            .dense(120)
            .relu()
            .dense(N_CLASSES))

Pipeline Implementation

We are using the scikit-learn's pipeline framework to handle various pipeline scenarios. See pipeline.py for details.

Once made, a pipeline can be trained and evaluated using the function below:

In [18]:
def train_evaluate(pipeline, epochs=5, samples_per_epoch=50000, train=(X_train, y_train), test=(X_valid, y_valid)):
    """
    Repeat the training for the epochs and evaluate the performance
    """
    X, y = train
    learning_curve = []
    for i in range(epochs):
        indices = np.random.choice(len(X), size=samples_per_epoch)
        pipeline.fit(X[indices], y[indices])
        scores = [pipeline.score(*train), pipeline.score(*test)]
        learning_curve.append([i, *scores])
        print("Epoch: {:>3} Train Score: {:.3f} Evaluation Score: {:.3f}".format(i, *scores))
    return np.array(learning_curve).T # (epochs, train scores, eval scores)

The First Performance

Let's train a network using the first network. This performance is our initial benchmark.

In [19]:
def resize_image(image, shape=INPUT_SHAPE[:2]):
    return cv2.resize(image, shape)

loader = lambda image_file: resize_image(load_image(image_file))
In [20]:
with Session() as session:
    functions = [loader]
    pipeline = build_pipeline(functions, session, make_network1(), make_adam(1.0e-3))
    train_evaluate(pipeline)
Epoch:   0 Train Score: 0.841 Evaluation Score: 0.822
Epoch:   1 Train Score: 0.939 Evaluation Score: 0.916
Epoch:   2 Train Score: 0.955 Evaluation Score: 0.932
Epoch:   3 Train Score: 0.891 Evaluation Score: 0.866
Epoch:   4 Train Score: 0.948 Evaluation Score: 0.925

Observation:

This proves the network is working properly. The performance is pretty good for the barebone network.

I can see a bit of overfitting. This is likely because the network is exposed to the same images over and over since I'm using 5 epochs (50K samples per epoch). At this moment, it is good to see the network is able to overfit and not showing high biases. The network can handle these images and able to learn from the data.

Preprocessing

Image Augmentation

As the training set has very skewed distribution, if I simply increases the epochs or samples per epoch, the network will overfit to the training set. Hence, we should generate more training data using image augmentation.

In [21]:
def random_brightness(image, ratio):
    """
    Randomly adjust brightness of the image.
    """
    # HSV (Hue, Saturation, Value) is also called HSB ('B' for Brightness).
    hsv = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
    brightness = np.float64(hsv[:, :, 2])
    brightness = brightness * (1.0 + np.random.uniform(-ratio, ratio))
    brightness[brightness>255] = 255
    brightness[brightness<0] = 0
    hsv[:, :, 2] = brightness
    return cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)


def random_rotation(image, angle):
    """
    Randomly rotate the image
    """
    if angle == 0:
        return image
    angle = np.random.uniform(-angle, angle)
    rows, cols = image.shape[:2]
    size = cols, rows
    center = cols/2, rows/2
    scale = 1.0
    rotation = cv2.getRotationMatrix2D(center, angle, scale)
    return cv2.warpAffine(image, rotation, size)


def random_translation(image, translation):
    """
    Randomly move the image
    """
    if translation == 0:
        return 0
    rows, cols = image.shape[:2]
    size = cols, rows
    x = np.random.uniform(-translation, translation)
    y = np.random.uniform(-translation, translation)
    trans = np.float32([[1,0,x],[0,1,y]])
    return cv2.warpAffine(image, trans, size)


def random_shear(image, shear):
    """
    Randomly distort the image
    """
    if shear == 0:
        return image
    rows, cols = image.shape[:2]
    size = cols, rows
    left, right, top, bottom = shear, cols - shear, shear, rows - shear
    dx = np.random.uniform(-shear, shear)
    dy = np.random.uniform(-shear, shear)
    p1 = np.float32([[left   , top],[right   , top   ],[left, bottom]])
    p2 = np.float32([[left+dx, top],[right+dx, top+dy],[left, bottom+dy]])
    move = cv2.getAffineTransform(p1,p2)
    return cv2.warpAffine(image, move, size)
    
    
def augment_image(image, brightness, angle, translation, shear):
    image = random_brightness(image, brightness)
    image = random_rotation(image, angle)
    image = random_translation(image, translation)
    image = random_shear(image, shear)
    return image
In [22]:
augmenter = lambda x: augment_image(x, brightness=0.7, angle=10, translation=5, shear=2)

show_images(sample_data[10:], cols=10) # original
for _ in range(5):
    show_images(sample_data[10:], cols=10, func=augmenter)
In [23]:
with Session() as session:
    functions = [loader, augmenter]
    pipeline = build_pipeline(functions, session, make_network1(), make_adam(1.0e-3))
    train_evaluate(pipeline)
Epoch:   0 Train Score: 0.199 Evaluation Score: 0.204
Epoch:   1 Train Score: 0.464 Evaluation Score: 0.460
Epoch:   2 Train Score: 0.574 Evaluation Score: 0.572
Epoch:   3 Train Score: 0.633 Evaluation Score: 0.628
Epoch:   4 Train Score: 0.674 Evaluation Score: 0.675

Obervation:

The hyper-perameters like brightness, rotation, translation, shear parameters are manually tuned by looking at the randomly altered images. If the alteration is too great, it is not realistic. The same way that horizontal flip is not included, too big change like rotating 90 degree should not be used.

The performance with the augmentation is much worse than without it. There are two possible reasons:

  1. the network is not robust to these changes (it was memorizing than generalizing)
  2. the training requires more epochs (it takes more time to train with larger data)

Let's see how other preprocessing can improve the performance first. I hope the normalization and other technique will make the learning easier for the network. Once that's done, I will use much bigger epochs to properly measure the performance.

Data Normalization

The below will test various normalization technique to see which one has the best performance.

In [24]:
normalizers = [('x - 127.5',              lambda x: x - 127.5), 
               ('x/127.5 - 1.0',          lambda x: x/127.5 - 1.0), 
               ('x/255.0 - 0.5',          lambda x: x/255.0 - 0.5),
               ('x - x.mean()',           lambda x: x - x.mean()),
               ('(x - x.mean())/x.std()', lambda x: (x - x.mean())/x.std())]
                
for name, normalizer in normalizers:
    print('Normalizer: {}'.format(name))
    with Session() as session:
        functions = [loader, augmenter, normalizer]
        pipeline = build_pipeline(functions, session, make_network1(), make_adam(1.0e-3))
        train_evaluate(pipeline)
    print()
Normalizer: x - 127.5
Epoch:   0 Train Score: 0.436 Evaluation Score: 0.426
Epoch:   1 Train Score: 0.594 Evaluation Score: 0.592
Epoch:   2 Train Score: 0.733 Evaluation Score: 0.719
Epoch:   3 Train Score: 0.736 Evaluation Score: 0.739
Epoch:   4 Train Score: 0.790 Evaluation Score: 0.782

Normalizer: x/127.5 - 1.0
Epoch:   0 Train Score: 0.593 Evaluation Score: 0.590
Epoch:   1 Train Score: 0.754 Evaluation Score: 0.754
Epoch:   2 Train Score: 0.811 Evaluation Score: 0.809
Epoch:   3 Train Score: 0.854 Evaluation Score: 0.851
Epoch:   4 Train Score: 0.849 Evaluation Score: 0.844

Normalizer: x/255.0 - 0.5
Epoch:   0 Train Score: 0.567 Evaluation Score: 0.558
Epoch:   1 Train Score: 0.680 Evaluation Score: 0.679
Epoch:   2 Train Score: 0.760 Evaluation Score: 0.750
Epoch:   3 Train Score: 0.786 Evaluation Score: 0.787
Epoch:   4 Train Score: 0.813 Evaluation Score: 0.812

Normalizer: x - x.mean()
Epoch:   0 Train Score: 0.515 Evaluation Score: 0.508
Epoch:   1 Train Score: 0.682 Evaluation Score: 0.674
Epoch:   2 Train Score: 0.755 Evaluation Score: 0.747
Epoch:   3 Train Score: 0.783 Evaluation Score: 0.778
Epoch:   4 Train Score: 0.820 Evaluation Score: 0.820

Normalizer: (x - x.mean())/x.std()
Epoch:   0 Train Score: 0.739 Evaluation Score: 0.735
Epoch:   1 Train Score: 0.831 Evaluation Score: 0.828
Epoch:   2 Train Score: 0.873 Evaluation Score: 0.866
Epoch:   3 Train Score: 0.901 Evaluation Score: 0.899
Epoch:   4 Train Score: 0.909 Evaluation Score: 0.900

Observaton:

The performance with any of the normalizations is better than without them. This clearly shows the importance of the normalizations. In this experiment, the normalization with (x-x.mean())/x.std() produced the best performance. The performance actually varies randomly at every run. So, it is not easy to say which one is better than what. In any case, (x-x.mean())/x.std() wins by far.

There are more techniques like using an average image of all training data, etc, which I may try later on. But for now, I will use the best performing normalization for the rest of the experiment.

In [25]:
normalizer = lambda x: (x - x.mean())/x.std()

Color Space

Now, we'll try difference color space to see if there is any performance gain.

Note: Gray scale has only one channel so it needs to be handled separately.

Color Code Reference:

In [26]:
# for Gray scale, we need to add the 3rd dimension back (1 channel) as it's expected by the network
converters = [('Gray', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2GRAY)[:, :, np.newaxis]),
              ('HSV', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2HSV)),
              ('HLS', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2HLS)),
              ('Lab', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2Lab)),
              ('Luv', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2Luv)),
              ('XYZ', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2XYZ)),
              ('Yrb', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2YCrCb)),
              ('YUV', lambda x: cv2.cvtColor(x, cv2.COLOR_RGB2YUV))]

GRAY_INPUT_SHAPE = (*INPUT_SHAPE[:2], 1)

for name, converter in converters:
    print('Color Space: {}'.format(name))
    with Session() as session:
        functions = [loader, augmenter, converter, normalizer]
        if name == 'Gray':
            network = make_network1(input_shape=GRAY_INPUT_SHAPE) # there is only one channel in gray scale
        else:
            network = make_network1()
        pipeline = build_pipeline(functions, session, network, make_adam(1.0e-3))
        train_evaluate(pipeline)
    print()
Color Space: Gray
Epoch:   0 Train Score: 0.734 Evaluation Score: 0.725
Epoch:   1 Train Score: 0.825 Evaluation Score: 0.822
Epoch:   2 Train Score: 0.854 Evaluation Score: 0.851
Epoch:   3 Train Score: 0.886 Evaluation Score: 0.887
Epoch:   4 Train Score: 0.903 Evaluation Score: 0.901

Color Space: HSV
Epoch:   0 Train Score: 0.521 Evaluation Score: 0.513
Epoch:   1 Train Score: 0.623 Evaluation Score: 0.618
Epoch:   2 Train Score: 0.691 Evaluation Score: 0.689
Epoch:   3 Train Score: 0.749 Evaluation Score: 0.743
Epoch:   4 Train Score: 0.785 Evaluation Score: 0.777

Color Space: HLS
Epoch:   0 Train Score: 0.471 Evaluation Score: 0.478
Epoch:   1 Train Score: 0.576 Evaluation Score: 0.573
Epoch:   2 Train Score: 0.652 Evaluation Score: 0.647
Epoch:   3 Train Score: 0.718 Evaluation Score: 0.719
Epoch:   4 Train Score: 0.770 Evaluation Score: 0.768

Color Space: Lab
Epoch:   0 Train Score: 0.583 Evaluation Score: 0.580
Epoch:   1 Train Score: 0.715 Evaluation Score: 0.715
Epoch:   2 Train Score: 0.778 Evaluation Score: 0.778
Epoch:   3 Train Score: 0.820 Evaluation Score: 0.811
Epoch:   4 Train Score: 0.848 Evaluation Score: 0.844

Color Space: Luv
Epoch:   0 Train Score: 0.578 Evaluation Score: 0.577
Epoch:   1 Train Score: 0.716 Evaluation Score: 0.712
Epoch:   2 Train Score: 0.775 Evaluation Score: 0.774
Epoch:   3 Train Score: 0.813 Evaluation Score: 0.810
Epoch:   4 Train Score: 0.844 Evaluation Score: 0.838

Color Space: XYZ
Epoch:   0 Train Score: 0.727 Evaluation Score: 0.720
Epoch:   1 Train Score: 0.831 Evaluation Score: 0.836
Epoch:   2 Train Score: 0.878 Evaluation Score: 0.877
Epoch:   3 Train Score: 0.902 Evaluation Score: 0.900
Epoch:   4 Train Score: 0.909 Evaluation Score: 0.899

Color Space: Yrb
Epoch:   0 Train Score: 0.563 Evaluation Score: 0.557
Epoch:   1 Train Score: 0.679 Evaluation Score: 0.673
Epoch:   2 Train Score: 0.756 Evaluation Score: 0.752
Epoch:   3 Train Score: 0.806 Evaluation Score: 0.800
Epoch:   4 Train Score: 0.839 Evaluation Score: 0.838

Color Space: YUV
Epoch:   0 Train Score: 0.589 Evaluation Score: 0.587
Epoch:   1 Train Score: 0.756 Evaluation Score: 0.753
Epoch:   2 Train Score: 0.805 Evaluation Score: 0.802
Epoch:   3 Train Score: 0.844 Evaluation Score: 0.841
Epoch:   4 Train Score: 0.841 Evaluation Score: 0.834

Observaton:

RGB (no conversion) is the best which surprised me. I was expecting the gray scale to be more efficient as the traffic signs are mostly about shapes not colors. The gray scale would have reduced the dimensionality from 3 color channels to 1, which would make the learning faster/easier. Apparently, that is not the case.

Also, I was thinking that the colors in traffic signs are more saturated than that of backgrounds (i.e., trees), and the color space like HSV and HLS might contribute to superior performance. This was not the case. Also, now that I saw the result, I realized that I should not assume anything about the background colors.

On a separate note, I noticed that whenever I ran this cell, the result seems slightly different. For example, gray scale or XYZ sometimes shows better performance than RGB (no conversion). This could be due to the randomness of image augmentation. But most of the times, RGB (no conversion) is the best. If I needed to analyse this further, I'd need to check the histogram of different channels and the performance for each color space.

But that's the kind of things the network should automatically figure out (i.e., automatic feature engineering). So, I should not mess with color space at least for now. It makes the pipeline simpler, too.

I'm done with preprocessing part.

In [27]:
preprocessors = [loader, augmenter, normalizer]

Model Improvement

I want to try the following to see if I can improve the performance while not causing any overfit.

  • more filters in conv layers
  • more neurons in dense layers
  • more conv layers
  • more dense layers
  • different activation like elu instead of relu
  • dropout

The below is to plot the learning curve.

In [28]:
def show_learning_curve(learning_curve):
    epochs, train, valid = learning_curve
    plt.figure(figsize=(10, 10))
    plt.plot(epochs, train, label='train')
    plt.plot(epochs, valid, label='validation')
    plt.title('Learning Curve')
    plt.ylabel('accuracy')
    plt.xlabel('epochs')
    plt.xticks(epochs)
    plt.legend(loc='center right')

These functions are for plotting and printing the confusion matrix.

In [29]:
def plot_confusion_matrix(cm):
    cm = [row/sum(row)   for row in cm]
    fig = plt.figure(figsize=(10, 10))
    ax = fig.add_subplot(111)
    cax = ax.matshow(cm, cmap=plt.cm.Oranges)
    fig.colorbar(cax)
    plt.title('Confusion Matrix')
    plt.xlabel('Predicted Class IDs')
    plt.ylabel('True Class IDs')
    plt.show()
    

def print_confusion_matrix(cm, sign_names=SIGN_NAMES):
    results = [(i, SIGN_NAMES[i], row[i]/sum(row)*100) for i, row in enumerate(cm)]
    accuracies = []
    for result in sorted(results, key=lambda x: -x[2]):
        print('{:>2} {:<50} {:6.2f}% {:>4}'.format(*result, sum(y_train==result[0])))
        accuracies.append(result[2])
    print('-'*50)
    print('Accuracy: Mean: {:.3f} Std: {:.3f}'.format(np.mean(accuracies), np.std(accuracies)))

Network 2

Doubling all filters in the convolutional layers and neurons in the dense layers.

In [30]:
def make_network2(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 12])  # <== doubled
            .max_pool()
            .relu()
            .conv([5, 5, 32])  # <== doubled
            .max_pool()
            .relu()
            .flatten()
            .dense(240) # <== doubled
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network2(), make_adam(1.0e-3))
    learning_curve = train_evaluate(pipeline)
    session.save('checkpoint/network2.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.853 Evaluation Score: 0.844
Epoch:   1 Train Score: 0.908 Evaluation Score: 0.908
Epoch:   2 Train Score: 0.937 Evaluation Score: 0.937
Epoch:   3 Train Score: 0.957 Evaluation Score: 0.954
Epoch:   4 Train Score: 0.951 Evaluation Score: 0.949
In [31]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network2())
    session.load('checkpoint/network2.ckpt')  
    pred = pipeline.predict(X_valid)
    
# examine confusionconfusion_matrixix
cm = confusion_matrix(y_valid, pred)
plot_confusion_matrix(cm)
print_confusion_matrix(cm)
14 Stop                                               100.00%  621
37 Go straight or left                                100.00%  167
12 Priority road                                       99.30% 1672
35 Ahead only                                          99.18%  955
10 No passing for vehicles over 3.5 metric tons        99.02% 1600
13 Yield                                               98.87% 1719
16 Vehicles over 3.5 metric tons prohibited            98.84%  334
22 Bumpy road                                          98.75%  310
17 No entry                                            98.67%  884
38 Keep right                                          98.10% 1648
42 End of no passing by vehicles over 3.5 metric tons  97.96%  191
 1 Speed limit (30km/h)                                97.79% 1767
25 Road work                                           97.71% 1194
 6 End of speed limit (80km/h)                         97.67%  334
34 Turn left ahead                                     97.67%  334
11 Right-of-way at the next intersection               97.40% 1051
39 Keep left                                           96.72%  239
33 Turn right ahead                                    96.45%  548
36 Go straight or right                                96.25%  310
31 Wild animals crossing                               96.23%  621
23 Slippery road                                       96.15%  406
26 Traffic signals                                     95.90%  478
 5 Speed limit (80km/h)                                95.78% 1481
28 Children crossing                                   95.45%  430
18 General caution                                     95.10%  955
15 No vehicles                                         93.80%  501
 9 No passing                                          93.67% 1170
40 Roundabout mandatory                                93.15%  287
 3 Speed limit (60km/h)                                92.36% 1122
41 End of no passing                                   91.84%  191
 8 Speed limit (120km/h)                               90.97% 1122
30 Beware of ice/snow                                  90.22%  358
 2 Speed limit (50km/h)                                89.76% 1791
29 Bicycles crossing                                   89.09%  215
19 Dangerous curve to the left                         88.37%  167
21 Double curve                                        88.06%  263
20 Dangerous curve to the right                        87.67%  287
 7 Speed limit (100km/h)                               87.41% 1146
 4 Speed limit (70km/h)                                85.89% 1576
32 End of all speed and passing limits                 85.71%  191
27 Pedestrians                                         79.59%  191
24 Road narrows on the right                           72.73%  215
 0 Speed limit (20km/h)                                60.47%  167
--------------------------------------------------
Accuracy: Mean: 93.064 Std: 7.605

Observation:

The performance improved. The training accuracy is slightly higher than the validation accuracy. It might be a sign of overfitting but I'll need to see by increasing the complexity of the network.

On a separate note, I could have tried changing layer by layer but changing all three did work so I'm ok with this.

The confusion matrix's mean accuracy is the sum of the mean accuracy for each class divided by the number of class. It is lower than overall accuracy indicating the larger classes are performing better (or the smaller classes are performing worse).

Network 3

Futher doubling all filters in the convolutional layers and neurons in the dense layers.

In [32]:
def make_network3(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24]) # <== doubled
            .max_pool()
            .relu()
            .conv([5, 5, 64]) # <== doubled
            .max_pool()
            .relu()
            .flatten()
            .dense(480)  # <== doubled
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(1.0e-3))
    learning_curve = train_evaluate(pipeline)
    session.save('checkpoint/network3.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.904 Evaluation Score: 0.900
Epoch:   1 Train Score: 0.949 Evaluation Score: 0.946
Epoch:   2 Train Score: 0.963 Evaluation Score: 0.959
Epoch:   3 Train Score: 0.968 Evaluation Score: 0.964
Epoch:   4 Train Score: 0.967 Evaluation Score: 0.961
In [33]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3())
    session.load('checkpoint/network3.ckpt')  
    pred = pipeline.predict(X_valid)
    
# examine confusionconfusion_matrixix
cm = confusion_matrix(y_valid, pred)
plot_confusion_matrix(cm)
print_confusion_matrix(cm)
12 Priority road                                      100.00% 1672
33 Turn right ahead                                   100.00%  548
25 Road work                                           99.67% 1194
17 No entry                                            99.56%  884
 8 Speed limit (120km/h)                               99.31% 1122
10 No passing for vehicles over 3.5 metric tons        99.27% 1600
15 No vehicles                                         99.22%  501
 6 End of speed limit (80km/h)                         98.84%  334
16 Vehicles over 3.5 metric tons prohibited            98.84%  334
22 Bumpy road                                          98.75%  310
26 Traffic signals                                     98.36%  478
28 Children crossing                                   98.18%  430
14 Stop                                                98.11%  621
38 Keep right                                          98.10% 1648
13 Yield                                               97.96% 1719
35 Ahead only                                          97.96%  955
 2 Speed limit (50km/h)                                97.82% 1791
37 Go straight or left                                 97.67%  167
 5 Speed limit (80km/h)                                97.36% 1481
40 Roundabout mandatory                                97.26%  287
30 Beware of ice/snow                                  96.74%  358
 1 Speed limit (30km/h)                                96.25% 1767
31 Wild animals crossing                               95.60%  621
 0 Speed limit (20km/h)                                95.35%  167
19 Dangerous curve to the left                         95.35%  167
 3 Speed limit (60km/h)                                95.14% 1122
39 Keep left                                           95.08%  239
 7 Speed limit (100km/h)                               94.22% 1146
 9 No passing                                          93.67% 1170
11 Right-of-way at the next intersection               93.31% 1051
23 Slippery road                                       93.27%  406
20 Dangerous curve to the right                        93.15%  287
 4 Speed limit (70km/h)                                93.07% 1576
18 General caution                                     93.06%  955
32 End of all speed and passing limits                 91.84%  191
41 End of no passing                                   91.84%  191
36 Go straight or right                                91.25%  310
27 Pedestrians                                         89.80%  191
29 Bicycles crossing                                   89.09%  215
42 End of no passing by vehicles over 3.5 metric tons  87.76%  191
24 Road narrows on the right                           87.27%  215
21 Double curve                                        85.07%  263
34 Turn left ahead                                     82.56%  334
--------------------------------------------------
Accuracy: Mean: 95.162 Std: 4.203

Observation:

The performance is better. It may be showing a slight overfitting. But I don't think we need to apply any regularization at this stage. I should rather try more epochs to see how far it can improve.

For almost all classes, the network is producing better than 90% accuracy, proving that increasing the network complexity is making it more robust.

Network 3 with More Epochs

In [34]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(1.0e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network3_epochs-20.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.891 Evaluation Score: 0.893
Epoch:   1 Train Score: 0.938 Evaluation Score: 0.937
Epoch:   2 Train Score: 0.958 Evaluation Score: 0.951
Epoch:   3 Train Score: 0.969 Evaluation Score: 0.969
Epoch:   4 Train Score: 0.954 Evaluation Score: 0.950
Epoch:   5 Train Score: 0.958 Evaluation Score: 0.953
Epoch:   6 Train Score: 0.972 Evaluation Score: 0.968
Epoch:   7 Train Score: 0.976 Evaluation Score: 0.973
Epoch:   8 Train Score: 0.970 Evaluation Score: 0.968
Epoch:   9 Train Score: 0.974 Evaluation Score: 0.966
Epoch:  10 Train Score: 0.968 Evaluation Score: 0.966
Epoch:  11 Train Score: 0.973 Evaluation Score: 0.970
Epoch:  12 Train Score: 0.971 Evaluation Score: 0.967
Epoch:  13 Train Score: 0.980 Evaluation Score: 0.979
Epoch:  14 Train Score: 0.977 Evaluation Score: 0.974
Epoch:  15 Train Score: 0.980 Evaluation Score: 0.974
Epoch:  16 Train Score: 0.978 Evaluation Score: 0.976
Epoch:  17 Train Score: 0.980 Evaluation Score: 0.978
Epoch:  18 Train Score: 0.984 Evaluation Score: 0.981
Epoch:  19 Train Score: 0.981 Evaluation Score: 0.974
In [35]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3())
    session.load('checkpoint/network3_epochs-20.ckpt')  
    pred = pipeline.predict(X_valid)
    
# examine confusionconfusion_matrixix
cm = confusion_matrix(y_valid, pred)
plot_confusion_matrix(cm)
print_confusion_matrix(cm)
 0 Speed limit (20km/h)                               100.00%  167
17 No entry                                           100.00%  884
33 Turn right ahead                                   100.00%  548
37 Go straight or left                                100.00%  167
39 Keep left                                          100.00%  239
41 End of no passing                                  100.00%  191
10 No passing for vehicles over 3.5 metric tons        99.76% 1600
25 Road work                                           99.67% 1194
 4 Speed limit (70km/h)                                99.50% 1576
14 Stop                                                99.37%  621
35 Ahead only                                          99.18%  955
 6 End of speed limit (80km/h)                         98.84%  334
34 Turn left ahead                                     98.84%  334
38 Keep right                                          98.82% 1648
18 General caution                                     98.78%  955
36 Go straight or right                                98.75%  310
13 Yield                                               98.64% 1719
 3 Speed limit (60km/h)                                98.61% 1122
26 Traffic signals                                     98.36%  478
 2 Speed limit (50km/h)                                98.26% 1791
12 Priority road                                       98.13% 1672
31 Wild animals crossing                               98.11%  621
 7 Speed limit (100km/h)                               97.96% 1146
11 Right-of-way at the next intersection               97.77% 1051
 9 No passing                                          97.67% 1170
22 Bumpy road                                          97.50%  310
28 Children crossing                                   97.27%  430
 8 Speed limit (120km/h)                               97.22% 1122
16 Vehicles over 3.5 metric tons prohibited            96.51%  334
29 Bicycles crossing                                   96.36%  215
42 End of no passing by vehicles over 3.5 metric tons  95.92%  191
40 Roundabout mandatory                                95.89%  287
 5 Speed limit (80km/h)                                95.51% 1481
23 Slippery road                                       95.19%  406
 1 Speed limit (30km/h)                                94.70% 1767
30 Beware of ice/snow                                  94.57%  358
32 End of all speed and passing limits                 93.88%  191
15 No vehicles                                         93.80%  501
20 Dangerous curve to the right                        93.15%  287
21 Double curve                                        91.04%  263
24 Road narrows on the right                           90.91%  215
19 Dangerous curve to the left                         90.70%  167
27 Pedestrians                                         89.80%  191
--------------------------------------------------
Accuracy: Mean: 97.092 Std: 2.785

Observation:

The performance did improve but not in the last several epochs.

100% accuracy is achieved for more classes. Also, the bottom performer is improving as well.

Network 3 with More Epochs and Lower Learning Rate

Let's try a lower learning rate with epochs=20.

In [36]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(0.5e-3)) # <== lower learning rate
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network3_epochs-20_lr-0.5e-3.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.821 Evaluation Score: 0.821
Epoch:   1 Train Score: 0.920 Evaluation Score: 0.919
Epoch:   2 Train Score: 0.932 Evaluation Score: 0.924
Epoch:   3 Train Score: 0.958 Evaluation Score: 0.952
Epoch:   4 Train Score: 0.975 Evaluation Score: 0.975
Epoch:   5 Train Score: 0.972 Evaluation Score: 0.969
Epoch:   6 Train Score: 0.978 Evaluation Score: 0.973
Epoch:   7 Train Score: 0.976 Evaluation Score: 0.973
Epoch:   8 Train Score: 0.984 Evaluation Score: 0.980
Epoch:   9 Train Score: 0.982 Evaluation Score: 0.979
Epoch:  10 Train Score: 0.988 Evaluation Score: 0.984
Epoch:  11 Train Score: 0.986 Evaluation Score: 0.978
Epoch:  12 Train Score: 0.987 Evaluation Score: 0.984
Epoch:  13 Train Score: 0.984 Evaluation Score: 0.980
Epoch:  14 Train Score: 0.988 Evaluation Score: 0.982
Epoch:  15 Train Score: 0.986 Evaluation Score: 0.981
Epoch:  16 Train Score: 0.987 Evaluation Score: 0.983
Epoch:  17 Train Score: 0.989 Evaluation Score: 0.987
Epoch:  18 Train Score: 0.988 Evaluation Score: 0.983
Epoch:  19 Train Score: 0.984 Evaluation Score: 0.984
In [37]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3())
    session.load('checkpoint/network3_epochs-20_lr-0.5e-3.ckpt')  
    pred = pipeline.predict(X_valid)
    
# examine confusionconfusion_matrixix
cm = confusion_matrix(y_valid, pred)
plot_confusion_matrix(cm)
print_confusion_matrix(cm)
12 Priority road                                      100.00% 1672
14 Stop                                               100.00%  621
22 Bumpy road                                         100.00%  310
30 Beware of ice/snow                                 100.00%  358
31 Wild animals crossing                              100.00%  621
32 End of all speed and passing limits                100.00%  191
33 Turn right ahead                                   100.00%  548
38 Keep right                                         100.00% 1648
10 No passing for vehicles over 3.5 metric tons        99.76% 1600
 9 No passing                                          99.67% 1170
11 Right-of-way at the next intersection               99.63% 1051
 7 Speed limit (100km/h)                               99.32% 1146
 3 Speed limit (60km/h)                                99.31% 1122
15 No vehicles                                         99.22%  501
26 Traffic signals                                     99.18%  478
13 Yield                                               98.87% 1719
 6 End of speed limit (80km/h)                         98.84%  334
16 Vehicles over 3.5 metric tons prohibited            98.84%  334
17 No entry                                            98.67%  884
20 Dangerous curve to the right                        98.63%  287
21 Double curve                                        98.51%  263
 5 Speed limit (80km/h)                                98.42% 1481
25 Road work                                           98.37% 1194
39 Keep left                                           98.36%  239
 4 Speed limit (70km/h)                                98.27% 1576
23 Slippery road                                       98.08%  406
 1 Speed limit (30km/h)                                98.01% 1767
 2 Speed limit (50km/h)                                97.82% 1791
 0 Speed limit (20km/h)                                97.67%  167
19 Dangerous curve to the left                         97.67%  167
 8 Speed limit (120km/h)                               97.57% 1122
28 Children crossing                                   97.27%  430
40 Roundabout mandatory                                97.26%  287
18 General caution                                     96.73%  955
34 Turn left ahead                                     96.51%  334
24 Road narrows on the right                           96.36%  215
29 Bicycles crossing                                   96.36%  215
41 End of no passing                                   95.92%  191
37 Go straight or left                                 95.35%  167
27 Pedestrians                                         93.88%  191
42 End of no passing by vehicles over 3.5 metric tons  89.80%  191
35 Ahead only                                          86.94%  955
36 Go straight or right                                82.50%  310
--------------------------------------------------
Accuracy: Mean: 97.525 Std: 3.443

Observation:

The performance is almost the same or slightly better. The learning curve looks much smoother. The average accuracy per class is slightly better, too. Overall, I believe the smaller learning rate was a worthy change.

Netowrk 3 Even Smaller Learning Rate

In [38]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(1.0e-4)) # <== lower learning rate
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network3_epochs-20_lr-1.0e-4.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.547 Evaluation Score: 0.543
Epoch:   1 Train Score: 0.718 Evaluation Score: 0.718
Epoch:   2 Train Score: 0.803 Evaluation Score: 0.797
Epoch:   3 Train Score: 0.847 Evaluation Score: 0.835
Epoch:   4 Train Score: 0.877 Evaluation Score: 0.875
Epoch:   5 Train Score: 0.907 Evaluation Score: 0.899
Epoch:   6 Train Score: 0.920 Evaluation Score: 0.915
Epoch:   7 Train Score: 0.934 Evaluation Score: 0.930
Epoch:   8 Train Score: 0.938 Evaluation Score: 0.931
Epoch:   9 Train Score: 0.944 Evaluation Score: 0.937
Epoch:  10 Train Score: 0.951 Evaluation Score: 0.945
Epoch:  11 Train Score: 0.958 Evaluation Score: 0.953
Epoch:  12 Train Score: 0.961 Evaluation Score: 0.955
Epoch:  13 Train Score: 0.964 Evaluation Score: 0.958
Epoch:  14 Train Score: 0.972 Evaluation Score: 0.965
Epoch:  15 Train Score: 0.972 Evaluation Score: 0.966
Epoch:  16 Train Score: 0.966 Evaluation Score: 0.965
Epoch:  17 Train Score: 0.976 Evaluation Score: 0.972
Epoch:  18 Train Score: 0.977 Evaluation Score: 0.974
Epoch:  19 Train Score: 0.979 Evaluation Score: 0.976
In [39]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3())
    session.load('checkpoint/network3_epochs-20_lr-1.0e-4.ckpt')  
    pred = pipeline.predict(X_valid)
    
# examine confusionconfusion_matrixix
cm = confusion_matrix(y_valid, pred)
plot_confusion_matrix(cm)
print_confusion_matrix(cm)
 6 End of speed limit (80km/h)                        100.00%  334
16 Vehicles over 3.5 metric tons prohibited           100.00%  334
17 No entry                                           100.00%  884
22 Bumpy road                                         100.00%  310
33 Turn right ahead                                   100.00%  548
42 End of no passing by vehicles over 3.5 metric tons 100.00%  191
25 Road work                                           99.67% 1194
14 Stop                                                99.37%  621
 3 Speed limit (60km/h)                                99.31% 1122
12 Priority road                                       99.30% 1672
38 Keep right                                          99.29% 1648
10 No passing for vehicles over 3.5 metric tons        99.27% 1600
15 No vehicles                                         99.22%  501
35 Ahead only                                          99.18%  955
13 Yield                                               99.09% 1719
34 Turn left ahead                                     98.84%  334
21 Double curve                                        98.51%  263
18 General caution                                     98.37%  955
39 Keep left                                           98.36%  239
 9 No passing                                          98.33% 1170
29 Bicycles crossing                                   98.18%  215
31 Wild animals crossing                               98.11%  621
32 End of all speed and passing limits                 97.96%  191
 0 Speed limit (20km/h)                                97.67%  167
36 Go straight or right                                97.50%  310
11 Right-of-way at the next intersection               97.03% 1051
 2 Speed limit (50km/h)                                96.95% 1791
 4 Speed limit (70km/h)                                96.04% 1576
27 Pedestrians                                         95.92%  191
41 End of no passing                                   95.92%  191
20 Dangerous curve to the right                        95.89%  287
40 Roundabout mandatory                                95.89%  287
 1 Speed limit (30km/h)                                95.81% 1767
19 Dangerous curve to the left                         95.35%  167
37 Go straight or left                                 95.35%  167
23 Slippery road                                       95.19%  406
26 Traffic signals                                     95.08%  478
28 Children crossing                                   94.55%  430
 7 Speed limit (100km/h)                               93.88% 1146
 5 Speed limit (80km/h)                                93.14% 1481
24 Road narrows on the right                           92.73%  215
 8 Speed limit (120km/h)                               92.36% 1122
30 Beware of ice/snow                                  91.30%  358
--------------------------------------------------
Accuracy: Mean: 97.300 Std: 2.342

Observation:

  • The performance is worse and it seems to show some overfitting in the end.
  • The learning curve is much more smoother.
  • The mean accuracy per class is worse and its standard devaition is bigger.

Let's stick with the previous learning rate for the time being.

Network 4

Let's try leaky ReLU (to avoid dead ReLU issue if any)

In [40]:
def make_network4(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24])
            .max_pool()
            .relu(leak_ratio=0.01) # <== leaky ReLU
            .conv([5, 5, 64])
            .max_pool()
            .relu(leak_ratio=0.01) # <== leaky ReLU
            .flatten()
            .dense(480)
            .relu(leak_ratio=0.01) # <== leaky ReLU
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network4(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network4.ckpt')

show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.859 Evaluation Score: 0.856
Epoch:   1 Train Score: 0.935 Evaluation Score: 0.930
Epoch:   2 Train Score: 0.955 Evaluation Score: 0.951
Epoch:   3 Train Score: 0.958 Evaluation Score: 0.960
Epoch:   4 Train Score: 0.970 Evaluation Score: 0.966
Epoch:   5 Train Score: 0.976 Evaluation Score: 0.974
Epoch:   6 Train Score: 0.978 Evaluation Score: 0.977
Epoch:   7 Train Score: 0.981 Evaluation Score: 0.981
Epoch:   8 Train Score: 0.981 Evaluation Score: 0.979
Epoch:   9 Train Score: 0.985 Evaluation Score: 0.984
Epoch:  10 Train Score: 0.988 Evaluation Score: 0.984
Epoch:  11 Train Score: 0.987 Evaluation Score: 0.984
Epoch:  12 Train Score: 0.990 Evaluation Score: 0.987
Epoch:  13 Train Score: 0.989 Evaluation Score: 0.987
Epoch:  14 Train Score: 0.991 Evaluation Score: 0.987
Epoch:  15 Train Score: 0.986 Evaluation Score: 0.983
Epoch:  16 Train Score: 0.990 Evaluation Score: 0.987
Epoch:  17 Train Score: 0.985 Evaluation Score: 0.983
Epoch:  18 Train Score: 0.990 Evaluation Score: 0.987
Epoch:  19 Train Score: 0.990 Evaluation Score: 0.986

Observation:

No improvment.

Network 5

ELU (Exponential Linear Unit) activation which is supposed to be faster to learn than ReLU.

Reference: http://www.picalike.com/blog/2015/11/28/relu-was-yesterday-tomorrow-comes-elu/

In [41]:
def make_network5(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24])
            .max_pool()
            .elu()              # <== ELU
            .conv([5, 5, 64])
            .max_pool()
            .elu()              # <== ELU
            .flatten()
            .dense(480)
            .elu()              # <== ELU
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network5(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network5.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.808 Evaluation Score: 0.803
Epoch:   1 Train Score: 0.889 Evaluation Score: 0.884
Epoch:   2 Train Score: 0.922 Evaluation Score: 0.918
Epoch:   3 Train Score: 0.941 Evaluation Score: 0.935
Epoch:   4 Train Score: 0.939 Evaluation Score: 0.937
Epoch:   5 Train Score: 0.952 Evaluation Score: 0.949
Epoch:   6 Train Score: 0.944 Evaluation Score: 0.943
Epoch:   7 Train Score: 0.963 Evaluation Score: 0.960
Epoch:   8 Train Score: 0.962 Evaluation Score: 0.959
Epoch:   9 Train Score: 0.964 Evaluation Score: 0.963
Epoch:  10 Train Score: 0.970 Evaluation Score: 0.965
Epoch:  11 Train Score: 0.973 Evaluation Score: 0.972
Epoch:  12 Train Score: 0.974 Evaluation Score: 0.969
Epoch:  13 Train Score: 0.972 Evaluation Score: 0.966
Epoch:  14 Train Score: 0.977 Evaluation Score: 0.976
Epoch:  15 Train Score: 0.978 Evaluation Score: 0.977
Epoch:  16 Train Score: 0.974 Evaluation Score: 0.970
Epoch:  17 Train Score: 0.979 Evaluation Score: 0.976
Epoch:  18 Train Score: 0.974 Evaluation Score: 0.971
Epoch:  19 Train Score: 0.980 Evaluation Score: 0.973

Observation:

The permance is worse. Also, it did not learn faster.

Network 6

Let's try smaller initial weight value.

In [42]:
def make_network6(input_shape=INPUT_SHAPE):
    return (NeuralNetwork(weight_sigma=0.01) # <== smaller weight sigma
            .input(input_shape)
            .conv([5, 5, 24])
            .max_pool()
            .relu()
            .conv([5, 5, 64])
            .max_pool()
            .relu()
            .flatten()
            .dense(480)
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network6(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network6.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.800 Evaluation Score: 0.796
Epoch:   1 Train Score: 0.910 Evaluation Score: 0.909
Epoch:   2 Train Score: 0.948 Evaluation Score: 0.944
Epoch:   3 Train Score: 0.956 Evaluation Score: 0.955
Epoch:   4 Train Score: 0.965 Evaluation Score: 0.958
Epoch:   5 Train Score: 0.965 Evaluation Score: 0.960
Epoch:   6 Train Score: 0.974 Evaluation Score: 0.973
Epoch:   7 Train Score: 0.978 Evaluation Score: 0.973
Epoch:   8 Train Score: 0.974 Evaluation Score: 0.969
Epoch:   9 Train Score: 0.970 Evaluation Score: 0.967
Epoch:  10 Train Score: 0.971 Evaluation Score: 0.968
Epoch:  11 Train Score: 0.983 Evaluation Score: 0.982
Epoch:  12 Train Score: 0.984 Evaluation Score: 0.983
Epoch:  13 Train Score: 0.987 Evaluation Score: 0.985
Epoch:  14 Train Score: 0.987 Evaluation Score: 0.985
Epoch:  15 Train Score: 0.985 Evaluation Score: 0.983
Epoch:  16 Train Score: 0.988 Evaluation Score: 0.985
Epoch:  17 Train Score: 0.989 Evaluation Score: 0.985
Epoch:  18 Train Score: 0.986 Evaluation Score: 0.984
Epoch:  19 Train Score: 0.987 Evaluation Score: 0.985

Observation:

Not an improvement - a bit worse.

Network 7

Adding one more dense layer.

In [43]:
def make_network7(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24])
            .max_pool()
            .relu()
            .conv([5, 5, 64])
            .max_pool()
            .relu()
            .flatten()
            .dense(480)
            .relu()
            .dense(240) # <== one more dense layer
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network7(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network7.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.815 Evaluation Score: 0.809
Epoch:   1 Train Score: 0.915 Evaluation Score: 0.917
Epoch:   2 Train Score: 0.926 Evaluation Score: 0.924
Epoch:   3 Train Score: 0.960 Evaluation Score: 0.962
Epoch:   4 Train Score: 0.965 Evaluation Score: 0.964
Epoch:   5 Train Score: 0.975 Evaluation Score: 0.973
Epoch:   6 Train Score: 0.967 Evaluation Score: 0.967
Epoch:   7 Train Score: 0.967 Evaluation Score: 0.962
Epoch:   8 Train Score: 0.980 Evaluation Score: 0.978
Epoch:   9 Train Score: 0.980 Evaluation Score: 0.977
Epoch:  10 Train Score: 0.975 Evaluation Score: 0.970
Epoch:  11 Train Score: 0.988 Evaluation Score: 0.984
Epoch:  12 Train Score: 0.984 Evaluation Score: 0.981
Epoch:  13 Train Score: 0.987 Evaluation Score: 0.981
Epoch:  14 Train Score: 0.978 Evaluation Score: 0.976
Epoch:  15 Train Score: 0.981 Evaluation Score: 0.975
Epoch:  16 Train Score: 0.989 Evaluation Score: 0.989
Epoch:  17 Train Score: 0.984 Evaluation Score: 0.983
Epoch:  18 Train Score: 0.985 Evaluation Score: 0.982
Epoch:  19 Train Score: 0.985 Evaluation Score: 0.982

Observation:

No improvment - a bit worse.

Network 8

The same as Network 3 but using MaxPooling after ReLU.

In [44]:
def make_network8(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24])
            .relu()     
            .max_pool() # <== after ReLU
            .conv([5, 5, 64])
            .relu()
            .max_pool() # <== after ReLU
            .flatten()
            .dense(480)
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network8(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network8.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.828 Evaluation Score: 0.824
Epoch:   1 Train Score: 0.916 Evaluation Score: 0.917
Epoch:   2 Train Score: 0.941 Evaluation Score: 0.940
Epoch:   3 Train Score: 0.963 Evaluation Score: 0.959
Epoch:   4 Train Score: 0.964 Evaluation Score: 0.960
Epoch:   5 Train Score: 0.977 Evaluation Score: 0.976
Epoch:   6 Train Score: 0.973 Evaluation Score: 0.971
Epoch:   7 Train Score: 0.984 Evaluation Score: 0.983
Epoch:   8 Train Score: 0.982 Evaluation Score: 0.981
Epoch:   9 Train Score: 0.981 Evaluation Score: 0.980
Epoch:  10 Train Score: 0.979 Evaluation Score: 0.977
Epoch:  11 Train Score: 0.980 Evaluation Score: 0.979
Epoch:  12 Train Score: 0.984 Evaluation Score: 0.980
Epoch:  13 Train Score: 0.985 Evaluation Score: 0.980
Epoch:  14 Train Score: 0.988 Evaluation Score: 0.988
Epoch:  15 Train Score: 0.990 Evaluation Score: 0.986
Epoch:  16 Train Score: 0.990 Evaluation Score: 0.988
Epoch:  17 Train Score: 0.990 Evaluation Score: 0.985
Epoch:  18 Train Score: 0.991 Evaluation Score: 0.986
Epoch:  19 Train Score: 0.989 Evaluation Score: 0.986

Observation:

No improvement - about the same.

Network 9

Let's try 3 convolutional layers.

In [45]:
def make_network9(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24])
            .max_pool()
            .relu()
            .conv([5, 5, 64])
            .max_pool()
            .relu()
            .conv([3, 3, 64])  # <= smaller kernel here (the image is small by here)
            .max_pool()
            .relu()
            .flatten()
            .dense(480)
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network9(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20)
    session.save('checkpoint/network9.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.848 Evaluation Score: 0.852
Epoch:   1 Train Score: 0.912 Evaluation Score: 0.907
Epoch:   2 Train Score: 0.945 Evaluation Score: 0.948
Epoch:   3 Train Score: 0.956 Evaluation Score: 0.959
Epoch:   4 Train Score: 0.966 Evaluation Score: 0.964
Epoch:   5 Train Score: 0.969 Evaluation Score: 0.961
Epoch:   6 Train Score: 0.973 Evaluation Score: 0.971
Epoch:   7 Train Score: 0.975 Evaluation Score: 0.976
Epoch:   8 Train Score: 0.980 Evaluation Score: 0.978
Epoch:   9 Train Score: 0.980 Evaluation Score: 0.974
Epoch:  10 Train Score: 0.965 Evaluation Score: 0.964
Epoch:  11 Train Score: 0.976 Evaluation Score: 0.974
Epoch:  12 Train Score: 0.977 Evaluation Score: 0.974
Epoch:  13 Train Score: 0.984 Evaluation Score: 0.982
Epoch:  14 Train Score: 0.980 Evaluation Score: 0.975
Epoch:  15 Train Score: 0.974 Evaluation Score: 0.970
Epoch:  16 Train Score: 0.982 Evaluation Score: 0.980
Epoch:  17 Train Score: 0.982 Evaluation Score: 0.979
Epoch:  18 Train Score: 0.983 Evaluation Score: 0.977
Epoch:  19 Train Score: 0.985 Evaluation Score: 0.981

Observation

No improvment - a bit worse.

Network 3 with Momentum Optimizer

In [46]:
for momentum in [0.7, 0.8, 0.9]:
    with Session() as session:
        print('Momentum: {}'.format(momentum))
        optimizer = tf.train.MomentumOptimizer(learning_rate=0.5e-3, momentum=momentum)
        pipeline = build_pipeline(preprocessors, session, make_network3(), optimizer)
        train_evaluate(pipeline, epochs=20)
        session.save('checkpoint/network3_momentum_{}.ckpt'.format(momentum))
        print()
Momentum: 0.7
Epoch:   0 Train Score: 0.362 Evaluation Score: 0.362
Epoch:   1 Train Score: 0.476 Evaluation Score: 0.469
Epoch:   2 Train Score: 0.551 Evaluation Score: 0.545
Epoch:   3 Train Score: 0.626 Evaluation Score: 0.624
Epoch:   4 Train Score: 0.674 Evaluation Score: 0.676
Epoch:   5 Train Score: 0.708 Evaluation Score: 0.708
Epoch:   6 Train Score: 0.745 Evaluation Score: 0.737
Epoch:   7 Train Score: 0.766 Evaluation Score: 0.765
Epoch:   8 Train Score: 0.791 Evaluation Score: 0.786
Epoch:   9 Train Score: 0.793 Evaluation Score: 0.791
Epoch:  10 Train Score: 0.824 Evaluation Score: 0.824
Epoch:  11 Train Score: 0.831 Evaluation Score: 0.827
Epoch:  12 Train Score: 0.847 Evaluation Score: 0.851
Epoch:  13 Train Score: 0.862 Evaluation Score: 0.860
Epoch:  14 Train Score: 0.868 Evaluation Score: 0.862
Epoch:  15 Train Score: 0.877 Evaluation Score: 0.871
Epoch:  16 Train Score: 0.876 Evaluation Score: 0.867
Epoch:  17 Train Score: 0.888 Evaluation Score: 0.878
Epoch:  18 Train Score: 0.896 Evaluation Score: 0.894
Epoch:  19 Train Score: 0.900 Evaluation Score: 0.894

Momentum: 0.8
Epoch:   0 Train Score: 0.435 Evaluation Score: 0.430
Epoch:   1 Train Score: 0.556 Evaluation Score: 0.554
Epoch:   2 Train Score: 0.637 Evaluation Score: 0.635
Epoch:   3 Train Score: 0.711 Evaluation Score: 0.700
Epoch:   4 Train Score: 0.753 Evaluation Score: 0.739
Epoch:   5 Train Score: 0.789 Evaluation Score: 0.779
Epoch:   6 Train Score: 0.806 Evaluation Score: 0.799
Epoch:   7 Train Score: 0.823 Evaluation Score: 0.816
Epoch:   8 Train Score: 0.847 Evaluation Score: 0.838
Epoch:   9 Train Score: 0.863 Evaluation Score: 0.861
Epoch:  10 Train Score: 0.876 Evaluation Score: 0.869
Epoch:  11 Train Score: 0.883 Evaluation Score: 0.878
Epoch:  12 Train Score: 0.892 Evaluation Score: 0.889
Epoch:  13 Train Score: 0.905 Evaluation Score: 0.896
Epoch:  14 Train Score: 0.908 Evaluation Score: 0.900
Epoch:  15 Train Score: 0.916 Evaluation Score: 0.908
Epoch:  16 Train Score: 0.908 Evaluation Score: 0.902
Epoch:  17 Train Score: 0.920 Evaluation Score: 0.916
Epoch:  18 Train Score: 0.922 Evaluation Score: 0.915
Epoch:  19 Train Score: 0.936 Evaluation Score: 0.932

Momentum: 0.9
Epoch:   0 Train Score: 0.535 Evaluation Score: 0.525
Epoch:   1 Train Score: 0.688 Evaluation Score: 0.693
Epoch:   2 Train Score: 0.773 Evaluation Score: 0.770
Epoch:   3 Train Score: 0.825 Evaluation Score: 0.819
Epoch:   4 Train Score: 0.858 Evaluation Score: 0.851
Epoch:   5 Train Score: 0.869 Evaluation Score: 0.871
Epoch:   6 Train Score: 0.902 Evaluation Score: 0.898
Epoch:   7 Train Score: 0.903 Evaluation Score: 0.908
Epoch:   8 Train Score: 0.924 Evaluation Score: 0.917
Epoch:   9 Train Score: 0.933 Evaluation Score: 0.931
Epoch:  10 Train Score: 0.942 Evaluation Score: 0.937
Epoch:  11 Train Score: 0.942 Evaluation Score: 0.936
Epoch:  12 Train Score: 0.952 Evaluation Score: 0.945
Epoch:  13 Train Score: 0.949 Evaluation Score: 0.945
Epoch:  14 Train Score: 0.954 Evaluation Score: 0.952
Epoch:  15 Train Score: 0.959 Evaluation Score: 0.947
Epoch:  16 Train Score: 0.952 Evaluation Score: 0.945
Epoch:  17 Train Score: 0.966 Evaluation Score: 0.964
Epoch:  18 Train Score: 0.967 Evaluation Score: 0.965
Epoch:  19 Train Score: 0.967 Evaluation Score: 0.962

Observation:

Got worse.

Network 3 with Balanced Class Distribution of Training Data

Will it help to have a balanced class distribution of training data?

In [47]:
def balance_distribution(X, y, size):
    X_balanced = []
    y_balanced = []
    for c in range(N_CLASSES):
        data = X[y==c]
        indices = np.random.choice(sum(y==c), size)
        X_balanced.extend(X[y==c][indices])
        y_balanced.extend(y[y==c][indices])
    return np.array(X_balanced), np.array(y_balanced)

X_balanced, y_balanced = balance_distribution(X_train, y_train, 3000)

show_class_distribution(y_balanced, 'Balanced Train Set')

Let's try the balanced data set with our best pipeline (Network 3 with learning rate = 0.5e-3)

In [48]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(0.5e-3))
    learning_curve = train_evaluate(pipeline, epochs=20, train=(X_balanced, y_balanced)) # <== using the balanced train set
    session.save('checkpoint/network3_with_balanced_data.ckpt')

show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.815 Evaluation Score: 0.756
Epoch:   1 Train Score: 0.911 Evaluation Score: 0.874
Epoch:   2 Train Score: 0.957 Evaluation Score: 0.938
Epoch:   3 Train Score: 0.962 Evaluation Score: 0.945
Epoch:   4 Train Score: 0.970 Evaluation Score: 0.949
Epoch:   5 Train Score: 0.976 Evaluation Score: 0.960
Epoch:   6 Train Score: 0.980 Evaluation Score: 0.969
Epoch:   7 Train Score: 0.981 Evaluation Score: 0.973
Epoch:   8 Train Score: 0.981 Evaluation Score: 0.968
Epoch:   9 Train Score: 0.986 Evaluation Score: 0.981
Epoch:  10 Train Score: 0.987 Evaluation Score: 0.980
Epoch:  11 Train Score: 0.990 Evaluation Score: 0.980
Epoch:  12 Train Score: 0.989 Evaluation Score: 0.979
Epoch:  13 Train Score: 0.983 Evaluation Score: 0.974
Epoch:  14 Train Score: 0.983 Evaluation Score: 0.975
Epoch:  15 Train Score: 0.988 Evaluation Score: 0.978
Epoch:  16 Train Score: 0.990 Evaluation Score: 0.981
Epoch:  17 Train Score: 0.989 Evaluation Score: 0.982
Epoch:  18 Train Score: 0.991 Evaluation Score: 0.983
Epoch:  19 Train Score: 0.989 Evaluation Score: 0.980

Observation:

The validation accuracy is much worse than before. This is likely because the distribution is different, indicating the network is learning the distribution which is different from the validation set. Assuming the test set has the same kind of validation (we should not check the test set at this stage), using the balance set may not be a good idea.

Instead, we should do more epochs so that minor classes are more visible to the network.

Network 3 with even more epochs

Let's just try with much more epochs.

In [49]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(0.5e-3)) 
    learning_curve = train_evaluate(pipeline, epochs=100)
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.840 Evaluation Score: 0.830
Epoch:   1 Train Score: 0.929 Evaluation Score: 0.929
Epoch:   2 Train Score: 0.955 Evaluation Score: 0.951
Epoch:   3 Train Score: 0.964 Evaluation Score: 0.961
Epoch:   4 Train Score: 0.973 Evaluation Score: 0.970
Epoch:   5 Train Score: 0.978 Evaluation Score: 0.972
Epoch:   6 Train Score: 0.973 Evaluation Score: 0.971
Epoch:   7 Train Score: 0.974 Evaluation Score: 0.971
Epoch:   8 Train Score: 0.986 Evaluation Score: 0.984
Epoch:   9 Train Score: 0.980 Evaluation Score: 0.976
Epoch:  10 Train Score: 0.984 Evaluation Score: 0.982
Epoch:  11 Train Score: 0.985 Evaluation Score: 0.979
Epoch:  12 Train Score: 0.989 Evaluation Score: 0.986
Epoch:  13 Train Score: 0.989 Evaluation Score: 0.983
Epoch:  14 Train Score: 0.989 Evaluation Score: 0.987
Epoch:  15 Train Score: 0.985 Evaluation Score: 0.982
Epoch:  16 Train Score: 0.989 Evaluation Score: 0.988
Epoch:  17 Train Score: 0.989 Evaluation Score: 0.988
Epoch:  18 Train Score: 0.991 Evaluation Score: 0.987
Epoch:  19 Train Score: 0.988 Evaluation Score: 0.983
Epoch:  20 Train Score: 0.991 Evaluation Score: 0.989
Epoch:  21 Train Score: 0.991 Evaluation Score: 0.989
Epoch:  22 Train Score: 0.992 Evaluation Score: 0.989
Epoch:  23 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  24 Train Score: 0.991 Evaluation Score: 0.991
Epoch:  25 Train Score: 0.990 Evaluation Score: 0.987
Epoch:  26 Train Score: 0.989 Evaluation Score: 0.987
Epoch:  27 Train Score: 0.990 Evaluation Score: 0.988
Epoch:  28 Train Score: 0.991 Evaluation Score: 0.989
Epoch:  29 Train Score: 0.992 Evaluation Score: 0.989
Epoch:  30 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  31 Train Score: 0.991 Evaluation Score: 0.985
Epoch:  32 Train Score: 0.993 Evaluation Score: 0.988
Epoch:  33 Train Score: 0.989 Evaluation Score: 0.985
Epoch:  34 Train Score: 0.991 Evaluation Score: 0.984
Epoch:  35 Train Score: 0.985 Evaluation Score: 0.982
Epoch:  36 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  37 Train Score: 0.993 Evaluation Score: 0.990
Epoch:  38 Train Score: 0.994 Evaluation Score: 0.993
Epoch:  39 Train Score: 0.992 Evaluation Score: 0.992
Epoch:  40 Train Score: 0.994 Evaluation Score: 0.992
Epoch:  41 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  42 Train Score: 0.995 Evaluation Score: 0.991
Epoch:  43 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  44 Train Score: 0.990 Evaluation Score: 0.986
Epoch:  45 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  46 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  47 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  48 Train Score: 0.993 Evaluation Score: 0.988
Epoch:  49 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  50 Train Score: 0.993 Evaluation Score: 0.988
Epoch:  51 Train Score: 0.993 Evaluation Score: 0.988
Epoch:  52 Train Score: 0.993 Evaluation Score: 0.990
Epoch:  53 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  54 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  55 Train Score: 0.993 Evaluation Score: 0.991
Epoch:  56 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  57 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  58 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  59 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  60 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  61 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  62 Train Score: 0.993 Evaluation Score: 0.989
Epoch:  63 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  64 Train Score: 0.988 Evaluation Score: 0.984
Epoch:  65 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  66 Train Score: 0.994 Evaluation Score: 0.992
Epoch:  67 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  68 Train Score: 0.994 Evaluation Score: 0.988
Epoch:  69 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  70 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  71 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  72 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  73 Train Score: 0.996 Evaluation Score: 0.991
Epoch:  74 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  75 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  76 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  77 Train Score: 0.994 Evaluation Score: 0.992
Epoch:  78 Train Score: 0.993 Evaluation Score: 0.989
Epoch:  79 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  80 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  81 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  82 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  83 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  84 Train Score: 0.995 Evaluation Score: 0.991
Epoch:  85 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  86 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  87 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  88 Train Score: 0.991 Evaluation Score: 0.989
Epoch:  89 Train Score: 0.993 Evaluation Score: 0.987
Epoch:  90 Train Score: 0.994 Evaluation Score: 0.989
Epoch:  91 Train Score: 0.995 Evaluation Score: 0.990
Epoch:  92 Train Score: 0.993 Evaluation Score: 0.989
Epoch:  93 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  94 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  95 Train Score: 0.994 Evaluation Score: 0.989
Epoch:  96 Train Score: 0.990 Evaluation Score: 0.987
Epoch:  97 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  98 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  99 Train Score: 0.993 Evaluation Score: 0.990

Observation:

It performs much better now but the last several epochs are not really helping for the network to learn. I should try smaller learning rate to see how it goes.

Network 3 with even lower learning rate (and more epochs)

In [50]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3(), make_adam(1.0e-4))
    learning_curve = train_evaluate(pipeline, epochs=500)
    session.save('checkpoint/network3_epochs-500_lr-1.0e-4.ckpt')    
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.568 Evaluation Score: 0.566
Epoch:   1 Train Score: 0.725 Evaluation Score: 0.718
Epoch:   2 Train Score: 0.792 Evaluation Score: 0.781
Epoch:   3 Train Score: 0.841 Evaluation Score: 0.839
Epoch:   4 Train Score: 0.877 Evaluation Score: 0.869
Epoch:   5 Train Score: 0.900 Evaluation Score: 0.890
Epoch:   6 Train Score: 0.910 Evaluation Score: 0.900
Epoch:   7 Train Score: 0.930 Evaluation Score: 0.924
Epoch:   8 Train Score: 0.939 Evaluation Score: 0.938
Epoch:   9 Train Score: 0.943 Evaluation Score: 0.939
Epoch:  10 Train Score: 0.952 Evaluation Score: 0.949
Epoch:  11 Train Score: 0.962 Evaluation Score: 0.957
Epoch:  12 Train Score: 0.967 Evaluation Score: 0.960
Epoch:  13 Train Score: 0.966 Evaluation Score: 0.962
Epoch:  14 Train Score: 0.970 Evaluation Score: 0.965
Epoch:  15 Train Score: 0.968 Evaluation Score: 0.961
Epoch:  16 Train Score: 0.971 Evaluation Score: 0.969
Epoch:  17 Train Score: 0.978 Evaluation Score: 0.970
Epoch:  18 Train Score: 0.982 Evaluation Score: 0.978
Epoch:  19 Train Score: 0.981 Evaluation Score: 0.976
Epoch:  20 Train Score: 0.979 Evaluation Score: 0.975
Epoch:  21 Train Score: 0.982 Evaluation Score: 0.978
Epoch:  22 Train Score: 0.983 Evaluation Score: 0.978
Epoch:  23 Train Score: 0.982 Evaluation Score: 0.978
Epoch:  24 Train Score: 0.986 Evaluation Score: 0.982
Epoch:  25 Train Score: 0.983 Evaluation Score: 0.982
Epoch:  26 Train Score: 0.989 Evaluation Score: 0.983
Epoch:  27 Train Score: 0.984 Evaluation Score: 0.980
Epoch:  28 Train Score: 0.987 Evaluation Score: 0.982
Epoch:  29 Train Score: 0.987 Evaluation Score: 0.985
Epoch:  30 Train Score: 0.989 Evaluation Score: 0.984
Epoch:  31 Train Score: 0.990 Evaluation Score: 0.988
Epoch:  32 Train Score: 0.990 Evaluation Score: 0.986
Epoch:  33 Train Score: 0.989 Evaluation Score: 0.985
Epoch:  34 Train Score: 0.990 Evaluation Score: 0.984
Epoch:  35 Train Score: 0.988 Evaluation Score: 0.984
Epoch:  36 Train Score: 0.990 Evaluation Score: 0.988
Epoch:  37 Train Score: 0.991 Evaluation Score: 0.984
Epoch:  38 Train Score: 0.991 Evaluation Score: 0.987
Epoch:  39 Train Score: 0.992 Evaluation Score: 0.990
Epoch:  40 Train Score: 0.993 Evaluation Score: 0.991
Epoch:  41 Train Score: 0.993 Evaluation Score: 0.989
Epoch:  42 Train Score: 0.993 Evaluation Score: 0.991
Epoch:  43 Train Score: 0.992 Evaluation Score: 0.990
Epoch:  44 Train Score: 0.993 Evaluation Score: 0.987
Epoch:  45 Train Score: 0.991 Evaluation Score: 0.988
Epoch:  46 Train Score: 0.995 Evaluation Score: 0.990
Epoch:  47 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  48 Train Score: 0.995 Evaluation Score: 0.990
Epoch:  49 Train Score: 0.995 Evaluation Score: 0.991
Epoch:  50 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  51 Train Score: 0.995 Evaluation Score: 0.994
Epoch:  52 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  53 Train Score: 0.993 Evaluation Score: 0.987
Epoch:  54 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  55 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  56 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  57 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  58 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  59 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  60 Train Score: 0.995 Evaluation Score: 0.991
Epoch:  61 Train Score: 0.996 Evaluation Score: 0.991
Epoch:  62 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  63 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  64 Train Score: 0.994 Evaluation Score: 0.991
Epoch:  65 Train Score: 0.994 Evaluation Score: 0.989
Epoch:  66 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  67 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  68 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  69 Train Score: 0.993 Evaluation Score: 0.990
Epoch:  70 Train Score: 0.995 Evaluation Score: 0.991
Epoch:  71 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  72 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  73 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  74 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  75 Train Score: 0.996 Evaluation Score: 0.992
Epoch:  76 Train Score: 0.996 Evaluation Score: 0.993
Epoch:  77 Train Score: 0.997 Evaluation Score: 0.991
Epoch:  78 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  79 Train Score: 0.998 Evaluation Score: 0.994
Epoch:  80 Train Score: 0.997 Evaluation Score: 0.996
Epoch:  81 Train Score: 0.997 Evaluation Score: 0.996
Epoch:  82 Train Score: 0.997 Evaluation Score: 0.993
Epoch:  83 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  84 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  85 Train Score: 0.997 Evaluation Score: 0.993
Epoch:  86 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  87 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  88 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  89 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  90 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  91 Train Score: 0.997 Evaluation Score: 0.993
Epoch:  92 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  93 Train Score: 0.998 Evaluation Score: 0.995
Epoch:  94 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  95 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  96 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  97 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  98 Train Score: 0.998 Evaluation Score: 0.995
Epoch:  99 Train Score: 0.997 Evaluation Score: 0.992
Epoch: 100 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 101 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 102 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 103 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 104 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 105 Train Score: 0.997 Evaluation Score: 0.995
Epoch: 106 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 107 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 108 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 109 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 110 Train Score: 0.997 Evaluation Score: 0.996
Epoch: 111 Train Score: 0.996 Evaluation Score: 0.991
Epoch: 112 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 113 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 114 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 115 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 116 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 117 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 118 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 119 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 120 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 121 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 122 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 123 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 124 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 125 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 126 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 127 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 128 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 129 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 130 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 131 Train Score: 0.998 Evaluation Score: 0.993
Epoch: 132 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 133 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 134 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 135 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 136 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 137 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 138 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 139 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 140 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 141 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 142 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 143 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 144 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 145 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 146 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 147 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 148 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 149 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 150 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 151 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 152 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 153 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 154 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 155 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 156 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 157 Train Score: 0.997 Evaluation Score: 0.993
Epoch: 158 Train Score: 0.998 Evaluation Score: 0.993
Epoch: 159 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 160 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 161 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 162 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 163 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 164 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 165 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 166 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 167 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 168 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 169 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 170 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 171 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 172 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 173 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 174 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 175 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 176 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 177 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 178 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 179 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 180 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 181 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 182 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 183 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 184 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 185 Train Score: 0.998 Evaluation Score: 0.993
Epoch: 186 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 187 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 188 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 189 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 190 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 191 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 192 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 193 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 194 Train Score: 0.999 Evaluation Score: 0.994
Epoch: 195 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 196 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 197 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 198 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 199 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 200 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 201 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 202 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 203 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 204 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 205 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 206 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 207 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 208 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 209 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 210 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 211 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 212 Train Score: 0.997 Evaluation Score: 0.995
Epoch: 213 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 214 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 215 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 216 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 217 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 218 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 219 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 220 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 221 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 222 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 223 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 224 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 225 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 226 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 227 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 228 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 229 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 230 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 231 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 232 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 233 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 234 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 235 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 236 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 237 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 238 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 239 Train Score: 0.999 Evaluation Score: 0.994
Epoch: 240 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 241 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 242 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 243 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 244 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 245 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 246 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 247 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 248 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 249 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 250 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 251 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 252 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 253 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 254 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 255 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 256 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 257 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 258 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 259 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 260 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 261 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 262 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 263 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 264 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 265 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 266 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 267 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 268 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 269 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 270 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 271 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 272 Train Score: 0.999 Evaluation Score: 0.994
Epoch: 273 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 274 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 275 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 276 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 277 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 278 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 279 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 280 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 281 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 282 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 283 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 284 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 285 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 286 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 287 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 288 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 289 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 290 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 291 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 292 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 293 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 294 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 295 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 296 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 297 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 298 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 299 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 300 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 301 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 302 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 303 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 304 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 305 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 306 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 307 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 308 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 309 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 310 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 311 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 312 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 313 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 314 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 315 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 316 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 317 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 318 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 319 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 320 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 321 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 322 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 323 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 324 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 325 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 326 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 327 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 328 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 329 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 330 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 331 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 332 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 333 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 334 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 335 Train Score: 1.000 Evaluation Score: 0.996
Epoch: 336 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 337 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 338 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 339 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 340 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 341 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 342 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 343 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 344 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 345 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 346 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 347 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 348 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 349 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 350 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 351 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 352 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 353 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 354 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 355 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 356 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 357 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 358 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 359 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 360 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 361 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 362 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 363 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 364 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 365 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 366 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 367 Train Score: 0.997 Evaluation Score: 0.994
Epoch: 368 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 369 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 370 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 371 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 372 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 373 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 374 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 375 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 376 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 377 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 378 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 379 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 380 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 381 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 382 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 383 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 384 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 385 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 386 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 387 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 388 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 389 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 390 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 391 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 392 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 393 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 394 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 395 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 396 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 397 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 398 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 399 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 400 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 401 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 402 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 403 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 404 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 405 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 406 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 407 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 408 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 409 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 410 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 411 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 412 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 413 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 414 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 415 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 416 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 417 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 418 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 419 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 420 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 421 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 422 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 423 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 424 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 425 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 426 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 427 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 428 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 429 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 430 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 431 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 432 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 433 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 434 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 435 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 436 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 437 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 438 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 439 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 440 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 441 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 442 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 443 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 444 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 445 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 446 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 447 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 448 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 449 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 450 Train Score: 0.999 Evaluation Score: 0.995
Epoch: 451 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 452 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 453 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 454 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 455 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 456 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 457 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 458 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 459 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 460 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 461 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 462 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 463 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 464 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 465 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 466 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 467 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 468 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 469 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 470 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 471 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 472 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 473 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 474 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 475 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 476 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 477 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 478 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 479 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 480 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 481 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 482 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 483 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 484 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 485 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 486 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 487 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 488 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 489 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 490 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 491 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 492 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 493 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 494 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 495 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 496 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 497 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 498 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 499 Train Score: 0.999 Evaluation Score: 0.997

Observation:

The performance has improved. It appears that 100 epochs are enough to achieve this performance.

Is this as good as it can get?

Network 10

Can we make the network more robust? How about a dropout?

In [51]:
def make_network10(input_shape=INPUT_SHAPE):
    return (NeuralNetwork()
            .input(input_shape)
            .conv([5, 5, 24])
            .max_pool()
            .relu()
            .conv([5, 5, 64])
            .max_pool()
            .relu()
            .dropout(keep_prob=0.5)
            .flatten()
            .dense(480)
            .relu()
            .dense(N_CLASSES))

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network10(), make_adam(1.0e-4))
    learning_curve = train_evaluate(pipeline, epochs=500)
    session.save('checkpoint/network10.ckpt')
    
show_learning_curve(learning_curve)
Epoch:   0 Train Score: 0.453 Evaluation Score: 0.451
Epoch:   1 Train Score: 0.652 Evaluation Score: 0.649
Epoch:   2 Train Score: 0.750 Evaluation Score: 0.741
Epoch:   3 Train Score: 0.825 Evaluation Score: 0.822
Epoch:   4 Train Score: 0.873 Evaluation Score: 0.867
Epoch:   5 Train Score: 0.899 Evaluation Score: 0.896
Epoch:   6 Train Score: 0.923 Evaluation Score: 0.917
Epoch:   7 Train Score: 0.939 Evaluation Score: 0.936
Epoch:   8 Train Score: 0.942 Evaluation Score: 0.939
Epoch:   9 Train Score: 0.952 Evaluation Score: 0.952
Epoch:  10 Train Score: 0.959 Evaluation Score: 0.957
Epoch:  11 Train Score: 0.960 Evaluation Score: 0.956
Epoch:  12 Train Score: 0.968 Evaluation Score: 0.967
Epoch:  13 Train Score: 0.970 Evaluation Score: 0.966
Epoch:  14 Train Score: 0.975 Evaluation Score: 0.973
Epoch:  15 Train Score: 0.976 Evaluation Score: 0.977
Epoch:  16 Train Score: 0.978 Evaluation Score: 0.977
Epoch:  17 Train Score: 0.982 Evaluation Score: 0.979
Epoch:  18 Train Score: 0.982 Evaluation Score: 0.981
Epoch:  19 Train Score: 0.983 Evaluation Score: 0.983
Epoch:  20 Train Score: 0.984 Evaluation Score: 0.984
Epoch:  21 Train Score: 0.986 Evaluation Score: 0.983
Epoch:  22 Train Score: 0.982 Evaluation Score: 0.980
Epoch:  23 Train Score: 0.988 Evaluation Score: 0.989
Epoch:  24 Train Score: 0.988 Evaluation Score: 0.987
Epoch:  25 Train Score: 0.988 Evaluation Score: 0.985
Epoch:  26 Train Score: 0.987 Evaluation Score: 0.985
Epoch:  27 Train Score: 0.990 Evaluation Score: 0.989
Epoch:  28 Train Score: 0.988 Evaluation Score: 0.988
Epoch:  29 Train Score: 0.989 Evaluation Score: 0.986
Epoch:  30 Train Score: 0.992 Evaluation Score: 0.991
Epoch:  31 Train Score: 0.992 Evaluation Score: 0.989
Epoch:  32 Train Score: 0.993 Evaluation Score: 0.990
Epoch:  33 Train Score: 0.991 Evaluation Score: 0.991
Epoch:  34 Train Score: 0.992 Evaluation Score: 0.990
Epoch:  35 Train Score: 0.992 Evaluation Score: 0.991
Epoch:  36 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  37 Train Score: 0.992 Evaluation Score: 0.988
Epoch:  38 Train Score: 0.993 Evaluation Score: 0.991
Epoch:  39 Train Score: 0.993 Evaluation Score: 0.992
Epoch:  40 Train Score: 0.993 Evaluation Score: 0.988
Epoch:  41 Train Score: 0.994 Evaluation Score: 0.992
Epoch:  42 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  43 Train Score: 0.993 Evaluation Score: 0.992
Epoch:  44 Train Score: 0.994 Evaluation Score: 0.990
Epoch:  45 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  46 Train Score: 0.994 Evaluation Score: 0.994
Epoch:  47 Train Score: 0.994 Evaluation Score: 0.992
Epoch:  48 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  49 Train Score: 0.994 Evaluation Score: 0.992
Epoch:  50 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  51 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  52 Train Score: 0.995 Evaluation Score: 0.994
Epoch:  53 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  54 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  55 Train Score: 0.995 Evaluation Score: 0.994
Epoch:  56 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  57 Train Score: 0.995 Evaluation Score: 0.991
Epoch:  58 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  59 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  60 Train Score: 0.995 Evaluation Score: 0.992
Epoch:  61 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  62 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  63 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  64 Train Score: 0.995 Evaluation Score: 0.995
Epoch:  65 Train Score: 0.995 Evaluation Score: 0.995
Epoch:  66 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  67 Train Score: 0.995 Evaluation Score: 0.993
Epoch:  68 Train Score: 0.997 Evaluation Score: 0.993
Epoch:  69 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  70 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  71 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  72 Train Score: 0.997 Evaluation Score: 0.996
Epoch:  73 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  74 Train Score: 0.997 Evaluation Score: 0.994
Epoch:  75 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  76 Train Score: 0.996 Evaluation Score: 0.994
Epoch:  77 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  78 Train Score: 0.997 Evaluation Score: 0.996
Epoch:  79 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  80 Train Score: 0.996 Evaluation Score: 0.995
Epoch:  81 Train Score: 0.997 Evaluation Score: 0.996
Epoch:  82 Train Score: 0.998 Evaluation Score: 0.997
Epoch:  83 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  84 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  85 Train Score: 0.998 Evaluation Score: 0.995
Epoch:  86 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  87 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  88 Train Score: 0.998 Evaluation Score: 0.996
Epoch:  89 Train Score: 0.998 Evaluation Score: 0.995
Epoch:  90 Train Score: 0.998 Evaluation Score: 0.996
Epoch:  91 Train Score: 0.998 Evaluation Score: 0.997
Epoch:  92 Train Score: 0.998 Evaluation Score: 0.996
Epoch:  93 Train Score: 0.997 Evaluation Score: 0.996
Epoch:  94 Train Score: 0.998 Evaluation Score: 0.996
Epoch:  95 Train Score: 0.997 Evaluation Score: 0.995
Epoch:  96 Train Score: 0.998 Evaluation Score: 0.997
Epoch:  97 Train Score: 0.998 Evaluation Score: 0.995
Epoch:  98 Train Score: 0.998 Evaluation Score: 0.996
Epoch:  99 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 100 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 101 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 102 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 103 Train Score: 0.997 Evaluation Score: 0.995
Epoch: 104 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 105 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 106 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 107 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 108 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 109 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 110 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 111 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 112 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 113 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 114 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 115 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 116 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 117 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 118 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 119 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 120 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 121 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 122 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 123 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 124 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 125 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 126 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 127 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 128 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 129 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 130 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 131 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 132 Train Score: 0.998 Evaluation Score: 0.998
Epoch: 133 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 134 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 135 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 136 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 137 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 138 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 139 Train Score: 0.998 Evaluation Score: 0.998
Epoch: 140 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 141 Train Score: 0.998 Evaluation Score: 0.994
Epoch: 142 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 143 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 144 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 145 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 146 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 147 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 148 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 149 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 150 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 151 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 152 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 153 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 154 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 155 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 156 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 157 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 158 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 159 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 160 Train Score: 0.998 Evaluation Score: 0.997
Epoch: 161 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 162 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 163 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 164 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 165 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 166 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 167 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 168 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 169 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 170 Train Score: 0.998 Evaluation Score: 0.996
Epoch: 171 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 172 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 173 Train Score: 0.998 Evaluation Score: 0.995
Epoch: 174 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 175 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 176 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 177 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 178 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 179 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 180 Train Score: 0.998 Evaluation Score: 0.998
Epoch: 181 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 182 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 183 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 184 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 185 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 186 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 187 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 188 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 189 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 190 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 191 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 192 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 193 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 194 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 195 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 196 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 197 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 198 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 199 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 200 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 201 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 202 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 203 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 204 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 205 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 206 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 207 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 208 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 209 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 210 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 211 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 212 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 213 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 214 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 215 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 216 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 217 Train Score: 0.998 Evaluation Score: 0.998
Epoch: 218 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 219 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 220 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 221 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 222 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 223 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 224 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 225 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 226 Train Score: 0.999 Evaluation Score: 0.996
Epoch: 227 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 228 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 229 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 230 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 231 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 232 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 233 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 234 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 235 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 236 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 237 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 238 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 239 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 240 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 241 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 242 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 243 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 244 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 245 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 246 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 247 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 248 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 249 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 250 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 251 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 252 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 253 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 254 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 255 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 256 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 257 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 258 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 259 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 260 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 261 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 262 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 263 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 264 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 265 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 266 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 267 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 268 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 269 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 270 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 271 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 272 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 273 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 274 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 275 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 276 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 277 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 278 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 279 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 280 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 281 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 282 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 283 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 284 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 285 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 286 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 287 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 288 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 289 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 290 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 291 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 292 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 293 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 294 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 295 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 296 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 297 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 298 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 299 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 300 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 301 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 302 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 303 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 304 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 305 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 306 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 307 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 308 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 309 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 310 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 311 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 312 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 313 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 314 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 315 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 316 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 317 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 318 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 319 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 320 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 321 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 322 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 323 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 324 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 325 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 326 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 327 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 328 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 329 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 330 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 331 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 332 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 333 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 334 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 335 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 336 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 337 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 338 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 339 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 340 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 341 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 342 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 343 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 344 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 345 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 346 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 347 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 348 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 349 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 350 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 351 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 352 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 353 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 354 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 355 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 356 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 357 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 358 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 359 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 360 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 361 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 362 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 363 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 364 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 365 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 366 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 367 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 368 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 369 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 370 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 371 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 372 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 373 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 374 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 375 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 376 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 377 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 378 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 379 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 380 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 381 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 382 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 383 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 384 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 385 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 386 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 387 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 388 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 389 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 390 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 391 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 392 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 393 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 394 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 395 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 396 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 397 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 398 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 399 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 400 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 401 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 402 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 403 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 404 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 405 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 406 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 407 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 408 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 409 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 410 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 411 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 412 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 413 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 414 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 415 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 416 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 417 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 418 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 419 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 420 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 421 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 422 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 423 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 424 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 425 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 426 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 427 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 428 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 429 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 430 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 431 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 432 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 433 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 434 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 435 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 436 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 437 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 438 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 439 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 440 Train Score: 1.000 Evaluation Score: 1.000
Epoch: 441 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 442 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 443 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 444 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 445 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 446 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 447 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 448 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 449 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 450 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 451 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 452 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 453 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 454 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 455 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 456 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 457 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 458 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 459 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 460 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 461 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 462 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 463 Train Score: 0.999 Evaluation Score: 0.997
Epoch: 464 Train Score: 1.000 Evaluation Score: 0.997
Epoch: 465 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 466 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 467 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 468 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 469 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 470 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 471 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 472 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 473 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 474 Train Score: 0.999 Evaluation Score: 0.999
Epoch: 475 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 476 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 477 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 478 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 479 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 480 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 481 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 482 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 483 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 484 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 485 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 486 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 487 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 488 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 489 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 490 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 491 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 492 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 493 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 494 Train Score: 0.999 Evaluation Score: 0.998
Epoch: 495 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 496 Train Score: 1.000 Evaluation Score: 0.998
Epoch: 497 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 498 Train Score: 1.000 Evaluation Score: 0.999
Epoch: 499 Train Score: 1.000 Evaluation Score: 0.998
In [52]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network10())
    session.load('checkpoint/network10.ckpt')  
    pred = pipeline.predict(X_valid)
    
# examine confusionconfusion_matrixix
cm = confusion_matrix(y_valid, pred)
plot_confusion_matrix(cm)
print_confusion_matrix(cm)
 0 Speed limit (20km/h)                               100.00%  167
 1 Speed limit (30km/h)                               100.00% 1767
 5 Speed limit (80km/h)                               100.00% 1481
 6 End of speed limit (80km/h)                        100.00%  334
 7 Speed limit (100km/h)                              100.00% 1146
10 No passing for vehicles over 3.5 metric tons       100.00% 1600
11 Right-of-way at the next intersection              100.00% 1051
12 Priority road                                      100.00% 1672
13 Yield                                              100.00% 1719
14 Stop                                               100.00%  621
15 No vehicles                                        100.00%  501
16 Vehicles over 3.5 metric tons prohibited           100.00%  334
19 Dangerous curve to the left                        100.00%  167
20 Dangerous curve to the right                       100.00%  287
22 Bumpy road                                         100.00%  310
24 Road narrows on the right                          100.00%  215
25 Road work                                          100.00% 1194
26 Traffic signals                                    100.00%  478
27 Pedestrians                                        100.00%  191
28 Children crossing                                  100.00%  430
29 Bicycles crossing                                  100.00%  215
30 Beware of ice/snow                                 100.00%  358
31 Wild animals crossing                              100.00%  621
32 End of all speed and passing limits                100.00%  191
33 Turn right ahead                                   100.00%  548
34 Turn left ahead                                    100.00%  334
35 Ahead only                                         100.00%  955
36 Go straight or right                               100.00%  310
37 Go straight or left                                100.00%  167
38 Keep right                                         100.00% 1648
39 Keep left                                          100.00%  239
40 Roundabout mandatory                               100.00%  287
41 End of no passing                                  100.00%  191
42 End of no passing by vehicles over 3.5 metric tons 100.00%  191
 2 Speed limit (50km/h)                                99.78% 1791
 9 No passing                                          99.67% 1170
18 General caution                                     99.59%  955
17 No entry                                            99.56%  884
 4 Speed limit (70km/h)                                99.50% 1576
 3 Speed limit (60km/h)                                99.31% 1122
 8 Speed limit (120km/h)                               99.31% 1122
23 Slippery road                                       99.04%  406
21 Double curve                                        97.01%  263
--------------------------------------------------
Accuracy: Mean: 99.832 Std: 0.491

Observation:

The validation performance is more stable now. 500 epochs is probably overkill.

Preprocessing (Again)

These are extra experiments to see if we can improve the pipeline further. I did experiment with these preprocessing before the network was fully trained and the effect was sort of random (sometimes good, other times bad). As such I discarded the idea.

But I'm re-doing it to see if it has positive effect after the network is fully trained.

The conclusion in short is that they don't work well. This is very likely due to the network trained without them. It seems abundantly clear in hindsight.

The below are a series of unfortunately experiments, proving how bad idea it was to change preprocessing after the training.

Image Enhancement

Taking a weighted average of the original image and the blurred image to make in order to smooth out the noises.

In [53]:
def enhance_image(image, ksize, weight):
    blurred = cv2.GaussianBlur(image, (ksize, ksize), 0)
    return cv2.addWeighted(image, weight, blurred, -weight, image.mean())
In [54]:
for ksize in [5, 7, 9, 11]:
    for weight in [4, 6, 8, 10]:
        print('Enhancer: k={} w={}'.format(ksize, weight))
        with Session() as session:
            enhancer = lambda x: enhance_image(x, ksize, weight)
            functions = [loader, augmenter, enhancer, normalizer]
            pipeline = build_pipeline(functions, session, make_network10())
            session.load('checkpoint/network10.ckpt')
            score = pipeline.score(X_valid, y_valid)
            print('Validation Score: {}'.format(score))
        print()
Enhancer: k=5 w=4
Validation Score: 0.708375

Enhancer: k=5 w=6
Validation Score: 0.69275

Enhancer: k=5 w=8
Validation Score: 0.67175

Enhancer: k=5 w=10
Validation Score: 0.66875

Enhancer: k=7 w=4
Validation Score: 0.793125

Enhancer: k=7 w=6
Validation Score: 0.767125

Enhancer: k=7 w=8
Validation Score: 0.755125

Enhancer: k=7 w=10
Validation Score: 0.7425

Enhancer: k=9 w=4
Validation Score: 0.823625

Enhancer: k=9 w=6
Validation Score: 0.796375

Enhancer: k=9 w=8
Validation Score: 0.77925

Enhancer: k=9 w=10
Validation Score: 0.757375

Enhancer: k=11 w=4
Validation Score: 0.87425

Enhancer: k=11 w=6
Validation Score: 0.85125

Enhancer: k=11 w=8
Validation Score: 0.8395

Enhancer: k=11 w=10
Validation Score: 0.82225

In [55]:
enhancer = lambda x: enhance_image(x, 9, 8)
In [56]:
show_images(sample_data[10:], cols=10)
show_images(sample_data[10:], cols=10, func=enhancer)

Observations:

I tried these earlier and the result was pretty random. Sometimes, it improves but not other times.

If I do this with the pre-trained network as shown above, the result is worse as the network is already tuned for images not using the enhancements.

Perhaps, I should try to use it during the training to see if it speed up the training or not.

However, as the network performs really well without this, my conclusion is not to use this filter at all.

Histogram Equalizaition

In [57]:
def equalizer(image):
    image = image.copy()
    for i in range(3):
        image[:, :, i] = cv2.equalizeHist(image[:, :, i])
    return image
In [58]:
show_images(sample_data[10:], cols=10)
show_images(sample_data[10:], cols=10, func=equalizer)
In [59]:
with Session() as session:
    functions = [loader, augmenter, equalizer, normalizer]
    pipeline = build_pipeline(functions, session, make_network10())
    session.load('checkpoint/network10.ckpt')
    score = pipeline.score(X_valid, y_valid)
    print('Validation Score: {:.3f}'.format(score))
Validation Score: 0.904

Observation:

If this was tried before the network was fully trained, it might have made the learning easier. Earlier, I played with this preprocessing before training but it was not producing better results than the enhancer. Maybe I will re-try that one day.

How about combine the histogram equalizer and the enhancer?

In [65]:
with Session() as session:
    functions = [loader, augmenter, equalizer, enhancer, normalizer]
    pipeline = build_pipeline(functions, session, make_network10())
    session.load('checkpoint/network10.ckpt')
    score = pipeline.score(X_valid, y_valid)
    print(score)
0.67775

Observaton:

The histogram equalizer + enhancer does not improve.

Min Max Normalization

In [62]:
def min_max_norm(image):
    return cv2.normalize(image, None, alpha=0, beta=255, norm_type=cv2.NORM_MINMAX)
In [63]:
show_images(sample_data[10:], cols=10)
show_images(sample_data[10:], cols=10, func=min_max_norm)
In [66]:
with Session() as session:
    functions = [loader, augmenter, min_max_norm, normalizer]
    pipeline = build_pipeline(functions, session, make_network10())
    session.load('checkpoint/network10.ckpt')
    score = pipeline.score(X_valid, y_valid)
    print(score)
0.99825

Observaton:

I tried Min-Max normalizer before fully training the network and did not get any good results.

This experiment here is just to check if it can do any good once the network is fully trained. It proves the otherwise.

Let's try the combination with the enhancer.

In [67]:
with Session() as session:
    functions = [loader, augmenter, min_max_norm, enhancer, normalizer]
    pipeline = build_pipeline(functions, session, make_network10())
    session.load('checkpoint/network10.ckpt')
    score = pipeline.score(X_valid, y_valid)
    print(score)
0.79925

Observation:

The same argument here. I don't need this.

Test Dataset

Test images are in one folder. So, we can simply load them as follows:

Test images do not have category folders but all are kept in one place with one label file.

data
 + Final_Test
    + Images
        + 00000.ppm
        + 00001.ppm
        + ...
        + GT-final_test.csv      # Extended annotations including class ids
        + GT-final_test.test.csv

I also downloaded GT-final_test.csv which contains extended annotations including class ids for test images.

In [68]:
TEST_IMAGE_DIR = 'data/Final_Test/Images'

# Note: GT-final_test.csv comes with class IDs (GT-final_test.test.csv does not)
test_df = pd.read_csv(os.path.join(TEST_IMAGE_DIR, 'GT-final_test.csv'), sep=';')
test_df['Filename'] = test_df['Filename'].apply(lambda x: os.path.join(TEST_IMAGE_DIR, x))
test_df.head()
Out[68]:
Filename Width Height Roi.X1 Roi.Y1 Roi.X2 Roi.Y2 ClassId
0 data/Final_Test/Images/00000.ppm 53 54 6 5 48 49 16
1 data/Final_Test/Images/00001.ppm 42 45 5 5 36 40 1
2 data/Final_Test/Images/00002.ppm 48 52 6 6 43 47 38
3 data/Final_Test/Images/00003.ppm 27 29 5 5 22 24 33
4 data/Final_Test/Images/00004.ppm 60 57 5 5 55 52 11
In [69]:
print("Number of test images: {:>5}".format(test_df.shape[0]))
Number of test images: 12630
In [70]:
X_test = test_df['Filename'].values
y_test = test_df['ClassId'].values
In [71]:
with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3())
    session.load('checkpoint/network10.ckpt')
    score = pipeline.score(X_test, y_test)
    print('Test Score: {}'.format(score))
Test Score: 0.958590657165479

Observation:

It's about 96% accuracy. For the simple network like this one, it did a really good work.

I probably need more complex network than this one but I'd need a better hardware to train one of those. Even with AWS g2 instance is not fast enough to run 500 epochs.

Testing with Random Images

In [72]:
X_new = np.array(glob.glob('images/sign*.jpg') + 
                 glob.glob('images/sign*.png'))
In [73]:
new_images = [plt.imread(path) for path in X_new]
In [74]:
print('-' * 80)
print('New Images for Random Testing')
print('-' * 80)

plt.figure(figsize=(15,5))
for i, image in enumerate(new_images):
    plt.subplot(2,len(X_new)//2,i+1)
    plt.imshow(image)
    plt.xticks([])
    plt.yticks([])    
plt.show()
--------------------------------------------------------------------------------
New Images for Random Testing
--------------------------------------------------------------------------------
In [75]:
print('getting top 5 results')

with Session() as session:
    pipeline = build_pipeline(preprocessors, session, make_network3())
    session.load('checkpoint/network10.ckpt')  
    prob = pipeline.predict_proba(X_new)
    estimator = pipeline.steps[-1][1]
    top_5_prob, top_5_pred = estimator.top_k_

print('done')
getting top 5 results
done
In [76]:
print('-' * 80)
print('Top 5 Predictions')
print('-' * 80)

for i, (preds, probs, image) in enumerate(zip(top_5_pred, top_5_prob, new_images)):
    plt.imshow(image)
    plt.xticks([])
    plt.yticks([])
    plt.show()
    for pred, prob in zip(preds.astype(int), probs):
        sign_name = SIGN_NAMES[pred]
        print('{:>5}: {:<50} ({:>14.10f}%)'.format(pred, sign_name, prob*100.0))
    print('-' * 80)    
--------------------------------------------------------------------------------
Top 5 Predictions
--------------------------------------------------------------------------------
   14: Stop                                               (100.0000000000%)
   13: Yield                                              (  0.0000000000%)
    1: Speed limit (30km/h)                               (  0.0000000000%)
   12: Priority road                                      (  0.0000000000%)
   25: Road work                                          (  0.0000000000%)
--------------------------------------------------------------------------------
    0: Speed limit (20km/h)                               (100.0000000000%)
    4: Speed limit (70km/h)                               (  0.0000000405%)
    1: Speed limit (30km/h)                               (  0.0000000012%)
   40: Roundabout mandatory                               (  0.0000000011%)
   24: Road narrows on the right                          (  0.0000000005%)
--------------------------------------------------------------------------------
    1: Speed limit (30km/h)                               ( 99.9801933765%)
    5: Speed limit (80km/h)                               (  0.0196605557%)
    2: Speed limit (50km/h)                               (  0.0001432495%)
    6: End of speed limit (80km/h)                        (  0.0000021335%)
    3: Speed limit (60km/h)                               (  0.0000000021%)
--------------------------------------------------------------------------------
    9: No passing                                         (100.0000000000%)
   15: No vehicles                                        (  0.0000000202%)
   10: No passing for vehicles over 3.5 metric tons       (  0.0000000012%)
    3: Speed limit (60km/h)                               (  0.0000000000%)
    2: Speed limit (50km/h)                               (  0.0000000000%)
--------------------------------------------------------------------------------
   35: Ahead only                                         (100.0000000000%)
   36: Go straight or right                               (  0.0000000000%)
   34: Turn left ahead                                    (  0.0000000000%)
   37: Go straight or left                                (  0.0000000000%)
   33: Turn right ahead                                   (  0.0000000000%)
--------------------------------------------------------------------------------
   25: Road work                                          ( 90.6243622303%)
   11: Right-of-way at the next intersection              (  8.8885203004%)
   27: Pedestrians                                        (  0.4870709963%)
   18: General caution                                    (  0.0000405782%)
   20: Dangerous curve to the right                       (  0.0000017497%)
--------------------------------------------------------------------------------
   30: Beware of ice/snow                                 ( 99.3382990360%)
   11: Right-of-way at the next intersection              (  0.6318126805%)
   21: Double curve                                       (  0.0165392499%)
   40: Roundabout mandatory                               (  0.0092216113%)
   29: Bicycles crossing                                  (  0.0038684269%)
--------------------------------------------------------------------------------
   34: Turn left ahead                                    (100.0000000000%)
   40: Roundabout mandatory                               (  0.0000000926%)
   36: Go straight or right                               (  0.0000000000%)
   35: Ahead only                                         (  0.0000000000%)
   38: Keep right                                         (  0.0000000000%)
--------------------------------------------------------------------------------
   40: Roundabout mandatory                               (100.0000000000%)
   37: Go straight or left                                (  0.0000000062%)
   34: Turn left ahead                                    (  0.0000000003%)
   38: Keep right                                         (  0.0000000000%)
   36: Go straight or right                               (  0.0000000000%)
--------------------------------------------------------------------------------
    5: Speed limit (80km/h)                               ( 99.9999761581%)
    8: Speed limit (120km/h)                              (  0.0000297359%)
    7: Speed limit (100km/h)                              (  0.0000014480%)
    1: Speed limit (30km/h)                               (  0.0000007021%)
    3: Speed limit (60km/h)                               (  0.0000000045%)
--------------------------------------------------------------------------------

Observation:

7 out of 10 are correct.

  • The pedestrian : This is actually not German traffic sign.
  • Speed limit (80km) : The model believed it was for 30km.
  • Speed limit (100km): The model believed it was for 80km.

I can understand why it did not identify the pedestrian correctly as it's not a German traffic sign. But it's quite similar to it. So, a human would have recognized this. This means that the machine learned only the shapes but not the concept, which is understandable from the way convolutional neural network works.

This also means that for every country / region, we'd need to train the classifier to recognize their traffic signs.

As for the speed limits (80km and 100km), I'm thinking it is may be due to the image distortion by the resizing operation. We may need a better way to resize images. But this is yet to be proven at this stage.

Conclusion

The use of pipelines were very effective during the experimentations. In the end, the traffic sign classifier works pretty well overall with the test set.

However, the network did not work as good with ramdom sample images from the internet.

Moreover, if a self-driving car needs to find traffic signs in public, it first needs to know where the traffic signs are. It's a chicken and egg problem.

Therefore, we will need an object recognition mechanism that scan across the image with sliding windows to find the candidate signs. This kind of detection mechanism is not covered in this project.

References: